Commit Graph

145 Commits

Author SHA1 Message Date
Alejandro Pedraza 8c67bfbcc6 Add parameter to stats API to skip retrieving Prometheus stats (#1871)
* Add parameter to stats API to skip retrieving Prometheus stats

Used by the dashboard to populate list of resources.

Fixes #1022

Signed-off-by: Alejandro Pedraza <alejandro.pedraza@gmail.com>

* Prometheus queries check results were being ignored
* Refactor verifyPromQueries() to also test when no prometheus queries
should be generated

* Add test for SkipStats=true

Includes adding ability to public.GenStatSummaryResponse to not generate
basicStats

* Fix previous test
2018-12-10 16:48:12 -08:00
Kevin Lingerfelt 0f8bcc9159
Controller: wait for caches to sync before opening listeners (#1958)
Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
2018-12-07 11:15:45 -08:00
Alex Leong 380ec52a39
Rework routes command to accept any resource (#1921)
We rework the routes command so that it can accept any Kubernetes resource, making it act much more similarly to the stat command.

Signed-off-by: Alex Leong <alex@buoyant.io>
2018-12-05 11:11:34 -08:00
Kevin Lingerfelt 37ae423bb3
Add linkerd- prefix to all objects in linkerd install (#1920)
Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
2018-12-04 15:41:47 -08:00
Andrew Seigner 37a5455445
Add filtering by job in stat, tap, top; fix panic (#1904)
Filtering by Kubernetes job was not supported. Also filtering by any unknown
type caused a panic.

Add filtering support by Kubernetes job, with special case mapping `job` to
`k8s_job`, to not conflict with Prometheus' job label.

Fix panic when unknown type specified as a `--from` or `--to` flag.

Fix `job` label from `linkerd-proxy` overwriting Prometheus `job` label at
collection time. This caused all metrics collected by proxy sidecars in
Kubernetes jobs to be collected into an incorrect Prometheus job, rather than
the expected `linkerd-proxy` Prometheus job.

Fix `unsupported resource type` tap error message incorrectly printing the
target resource rather than the destination.

Set `--controller-log-level debug` in `install_test.go` for easier debugging.

Expose `slow-cooker`'s metrics via a k8s service in the tap integration test, to
validate proxy requests with a job as destination.

Fixes #1872
Part of #627

Signed-off-by: Andrew Seigner <siggy@buoyant.io>
2018-12-03 15:34:49 -08:00
Risha Mars f8583df4db
Add ListServices to controller public api (#1876)
Add a barebones ListServices endpoint, in support of autocomplete for services.
As we develop service profiles, this endpoint could probably be used to describe
more aspects of services (like, if there were some way to check whether a
service profile was enabled or not).

Accessible from the web UI via http://localhost:8084/api/services
2018-11-27 11:34:47 -08:00
Alex Leong 73836f05cf
Update proxy version and use canonicalized dst (#1866)
The `linkerd` routes command only supports outbound metrics queries (i.e. ones with the `--from` flag).  Inbound queries (i.e. ones without the `--from` flag) never return any metrics.

We update the proxy version and use the new canonicalized form for dst labels to gain support for inbound metrics as well.

Signed-off-by: Alex Leong <alex@buoyant.io>
2018-11-26 17:20:07 -08:00
Alex Leong 7a7f6b6ecb
Add TopRoutes method the the public api and route CLI command to consume it (#1860)
Add a routes command which displays per-route stats for services that have service profiles defined.

This change has three parts:
* A new public-api RPC called `TopRoutes` which serves per-route stat data about a service
* An implementation of TopRoutes in the public-api service.  This implementation reads per-route data from Prometheus.  This is very similar to how the StatSummaries RPC and much of the code was able to be refactored and shared.
* A new CLI command called `routes` which displays the per-route data in a tabular or json format.  This is very similar to the `stat` command and much of the code was able to be refactored and shared.

Note that as of the currently targeted proxy version, only outbound route stats are supported so the `--from` flag must be included in order to see data.  This restriction will be lifted in an upcoming change once we add support for inbound route stats as well.

Signed-off-by: Alex Leong <alex@buoyant.io>
2018-11-19 12:20:30 -08:00
Kevin Leimkuhler c68693e820
Fix stat filtering for `--from` queries (#1856)
# Problem
When we add a `--from` query to `linkerd stat au` we get more rows than if we would have just run `linkerd stat au`.

Adding a `--from` causes an extra row to be added, and the named authority to be ignored (this is the result we would have expected when running `linkerd stat au -n emojivoto --from deploy/web`).

# Solution
Destination query labels are now appended to `labels` so that those labels can be filtered on.

# Validation
Tests have been updated to reflect the expected expected destination labels now appended in `--from` queries.

Fixes #1766 

Signed-off-by: Kevin Leimkuhler <kevin@kleimkuhler.com>
2018-11-14 10:52:27 -08:00
Alejandro Pedraza 37bc8a69db Added support for json output in `linkerd stat` (#1749)
Added support for json output in `linkerd stat` through a new (-o|--output)=json option.

Fixes #1417

Signed-off-by: Alejandro Pedraza <alejandro.pedraza@gmail.com>
2018-10-15 14:10:48 -07:00
Risha Mars 31a396b631
Fix incorrect test wording (#1767) 2018-10-15 12:07:06 -07:00
Alex Leong 1fe19bf3ce
Add ServiceProfile support to k8s utilities (#1758)
Updates to the Kubernetes utility code in `/controller/k8s` to support interacting with ServiceProfiles.

This makes use of the code generated client added in #1752 

Signed-off-by: Alex Leong <alex@buoyant.io>
2018-10-12 09:35:11 -07:00
Alena Varkockova 5a853e8990 Use ListPods always for data plane HC (#1701)
* Use ListPods always for data plane HC
* Missing changes in grpc_server.go
* Address review comments
* Read proxy version from spec

Signed-off-by: Alena Varkockova <varkockova.a@gmail.com>
2018-10-02 11:45:01 -07:00
Kevin Lingerfelt b5ff29c8aa
Add data plane check to validate proxy version (#1574)
Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
2018-09-04 15:22:38 -07:00
Kevin Lingerfelt e97be1f5da
Move all healthcheck-related code to pkg/healthcheck (#1492)
* Move all healthcheck-related code to pkg/healthcheck
* Fix failed check formatting
* Better version check wording

Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
2018-08-20 16:50:22 -07:00
Kevin Lingerfelt 00a0572098
Better CLI error messages when control plane is unavailable (#1428)
Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
2018-08-09 15:40:41 -07:00
Kevin Lingerfelt bd19e8aaff
Update prometheus to only scrape proxies in the same mesh (#1402)
Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
2018-08-06 12:05:55 -07:00
Alex Leong 3e1f35913b
Read all bytes of message length header (#1394)
The `reader.Read` method only reads as many bytes as are currently available from reader.  When reading the 4 byte message length header, if not all 4 of those bytes are available, `Read` will only read the available bytes and return.  This causes alignment issues when the message body is read and there are still unread header bytes in the reader.  These bytes will appear at the beginning of the message body and cause a crash when the message is unmarshalled.

Use `io.ReadFull` to ensure that we read all 4 of the message length header bytes.

Fixes #1287 

Signed-off-by: Alex Leong <alex@buoyant.io>
2018-08-02 10:45:49 -07:00
Risha Mars fef896011f Add more filters to the web UI tap form (#1371)
* Update ant to 3.7.2
* Add autocomplete of namespaces/resources to Tap in web ui
  * Add form fields for authority/path/method/rps/scheme
  * Add the ability to clear error messages to the error banner
* Add error listener to ws object
2018-07-31 15:48:53 -07:00
Kevin Lingerfelt 4b9700933a
Update prometheus labels to match k8s resource names (#1355)
Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
2018-07-23 15:45:05 -07:00
Kevin Lingerfelt e5cce1abaf
Rename CLI from conduit to linkerd (#1312)
* Rename CLI binary
* Update integration tests for new binary name
* Rename --conduit-namespace flag, change default ns
* Rename occurrences of conduit in rest of CLI
* Rename inject and install components
* Remove conduit occurrences in docker files
* Additional miscellaneous cleanup
* Move protobuf definitions to linkerd2 package
* Rename conduit.io labels to use linkerd.io
* Rename conduit-managed segment to linkerd-managed
* Fix conduit references in web project

Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
2018-07-12 17:14:07 -07:00
Oliver Gould 941cad4a9c
Migrate build infrastructure to linkerd2 (#1298)
This PR begins to migrate Conduit to Linkerd2:
* The proxy has been completely removed from this repo, and is now located at
  github.com/linkerd/linkerd2-proxy.
* A `Dockerfile-proxy` has been added to fetch the most-recently published proxy
  binary from build.l5d.io.
* Proxy-specific protobuf bindings have been moved to
  github.com/linkerd/linkerd2-proxy-api.
* All docker images now use the gcr.io/linkerd-io registry.
* `inject` now uses `LINKERD2_PROXY_` environment variables
* Go paths have been updated to reflect the new (future) repo location.
2018-07-09 15:38:38 -07:00
Risha Mars 9050b2d312
Fix authority stat queries when a --from flag is used (#1289)
* Fix bug where we were using dst_authorities as a group by instead of authorities
* Add test to make sure we don't dst_authorities

Previously, we were only checking to make sure we didn't add 
dst_authorities in the query labels in promDstQueryLabels but we 
weren't checking the groupBy labels in promDstGroupByLabelNames - 
this caused us to try to query for dst_authorities when a --from 
query was sent. There are no dst_authorities, so there would be no 
named results.
2018-07-06 17:29:08 -07:00
Kevin Lingerfelt 693acdbf26
Update ListPods endpoint to return all pod owner types (#1275)
Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
2018-07-05 15:14:16 -07:00
Risha Mars ba2e13c731
Small tweaks to error modal, add Reason to api error response (#1246)
- Add Reason to the error data passed from the api
- Rewrite error logic in the UI to try to make it clearer
- Show 0/0 pods meshed instead of 0/0 pods meshed (N/A) if 0 pods are meshed
2018-07-03 17:14:27 -07:00
Risha Mars 2002a8ba50
Add more tests for the stat summary endpoint --from flags (#1237)
Also add dst_ labels in the metrics we mock, so we can do --from queries with results.
2018-07-03 14:30:15 -07:00
Risha Mars 8ebc969d2f
Fix bug where we wouldn't run stat table assertions if we expected 0 results (#1235)
I realized that our stat summary expectation checker would only check the actual
proto responses against the expectations if the expectations were non-empty.

Problem
If we expected empty results and the api returned actual results, we never actually 
check those results against the expectations.

The bug can be reproduced by replacing any nonzero metric we expect in 
expectedResponse with expectedResponse: genEmptyResponse() 
The tests on master will still pass.

Solution
Remove this line and ensure we get the expected number of stat tables.
2018-06-29 14:23:14 -07:00
Risha Mars 5ed7fc563c
Add controller component pod uptimes to the ServiceMesh page (#1205)
- Return pod uptimes from the GetPods endpoint
- Adds filtering by namespace to api.GetPods
- Adds a --namespace filter to conduit get pods
- Adds pod uptimes to the controller component toolitps on the ServiceMesh page
- Moves the ServiceMesh page back to using /api/pods
2018-06-28 15:42:00 -07:00
Risha Mars 5963b2ac24
Better format empty errors (#1202) 2018-06-28 14:52:04 -07:00
Risha Mars 68586fe697
Add the ability to query stats by authority (#1181)
Adds the ability to query by a new non-kubernetes resource type, "authorities",
in the StatSummary api.

This includes an extensive refactor of stat_summary.go to deal with non-kubernetes 
resource types.

- Add documentation to Resource in the public api so we can use it for authority
- Handle non-k8s resource requests in the StatSummary endpoint
- Rewrite stat summary fetching and parsing to handle non-k8s resources
- keys stat summary metric handling by Resource instead of a generated string
- Adds authority to the CLI
- Adds /authorities to the Web UI
- Adds some more stat integration and unit tests
2018-06-28 14:31:44 -07:00
Kevin Lingerfelt 682b0274b5
Add controller admin servers and readiness probes (#1168)
* Add controller admin servers and readiness probes
* Tweak readiness probes to be more sane
* Refactor based on review feedback

Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
2018-06-20 17:32:44 -07:00
Risha Mars 46c99febf2
Don't panic on stats that aren't included in StatAllResourceTypes (#1154)
Problem
`conduit stat` would cause a panic for any resource that wasn't in the list 
of StatAllResourceTypes
This bug was introduced by https://github.com/runconduit/conduit/pull/1088/files

Solution
Fix writeStatsToBuffer to not depend on what resources are in StatAllResourceTypes
Also adds a unit test and integration test for `conduit stat ns`
2018-06-19 17:00:16 -07:00
Risha Mars e2c2f19d2c
Propagate errors in conduit containers to the api (#1117)
- It would be nice to display container errors in the UI. This PR gets the pod's container 
statuses and returns them in the public api

- Also add a terminationMessagePolicy to conduit's inject so that we can capture the 
proxy's error messages if it terminates
2018-06-14 16:22:31 -07:00
Kevin Lingerfelt 13aaa82c95
Allow k8s API clients to watch a subset of resources (#1118)
* Allow k8s API clients to watch a subset of resources
* Sort resources

Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
2018-06-14 11:09:01 -07:00
Kevin Lingerfelt 9f1df963e9
Move controller/util and web/util packages to pkg (#1109)
Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
2018-06-13 11:25:56 -07:00
Kevin Lingerfelt 6e66f6d662
Rename Lister to API and expose informers as well as listers (#1072)
Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
2018-06-12 10:27:55 -07:00
Risha Mars 7d4c4aa290
CLI: print resources in the same order every time stat all is run (#1088)
Previously, in conduit stat all we would just print the map of stat results, which 
resulted in the order in which stats were displayed varying between prints.

Fix:
Define an array, k8s.StatAllResourceTypes and use the order in this array to print 
the map; ensuring a consistent print order every time the command is run.
2018-06-08 15:02:17 -07:00
Ivan Sim 11d1d55632 Filter out failed and completed pods from stats summary result (#1010) (#1065)
Both the conduit stat command and web UI are showing failed and completed pods.
This change filters out those pods before returning the result to the client.

Fixes #1010

Signed-off-by: Ivan Sim <ihcsim@gmail.com>
2018-06-05 13:19:48 -07:00
Kevin Lingerfelt ec2433e9bd
Update controller to use 'tls' metric label (#1044)
* Update controller to use 'tls' metric label
* Fix meshed column formatter

Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
2018-06-01 16:44:33 -07:00
Risha Mars ffabdefc6c
Add queries to prometheus to determine number of fully meshed requests (#983)
- Update the `response_total` prometheus query of the StatSummary endpoint to also
break queries out by a `meshed` label. 
- Add a 'Secured' column to the web UI/CLI stat displays, which indicate the percentage of traffic
starting and ending in the mesh

This meshed label is used in the CLI/Web UI to display a column of the percentage of traffic that
starts/ends in the mesh. (Which is a proxy indicator for whether that traffic is 'secured' when we
add TLS by default for intra mesh requests).

The `meshed` label is not yet added anywhere, so until it is supplied by the proxy, all traffic will
show up as 0% secured in the web/CLI.
2018-05-24 11:05:09 -07:00
Andrew Seigner 84e6eb5c87
Fix nil pointer dereference in StatSummary (#991)
The StatSummary endpoint was dereferencing
StatSummaryRequest.Selector.Resource, causing a panic when it received
an empty request.

Fix StatSummary to use the nil-friendly
StatSummaryRequest.GetSelector().GetResource() methods, and add a test
to validate.

Signed-off-by: Andrew Seigner <siggy@buoyant.io>
2018-05-23 13:21:49 -07:00
Risha Mars 1e6434f6de
Fix bug in the public-api where conduit stat params were ignored (#971)
* Fix bug where we were dropping parts of the StatSummaryRequest
* Add tests for prometheus query strings and for failed cases

Problem
In #928 I rewrote the stat api to handle 'all' as a resource type. To query for all resource types, 
we would copy the Resource, LabelSelector and TimeWindow of the original request, and then 
go through all the resource types and set Resource.Type for each resource we wanted to get.
The bug is that while we copy over some fields of the original request, we didn't copy over all 
of them - namely Resource.Name and the Outbound resource. So the Stat endpoint would 
ignore any --to or --from flags, and would ignore requests for a specific named resource.

Solution
Copy over all fields from the request.

I've also added tests for this case. In this process I've refactored the stat_summary_test code 
to make it a bit easier to read/use.
2018-05-18 16:06:06 -07:00
Risha Mars b8dc83f9d2
Modify the Stat API to handle requests for resource type "all" (#928)
Allow the Stat endpoint in the public-api to accept requests for resourceType "all".

Currently, this queries Pods, Deployments, RCs and Services, but can be modified 
to query other resources as well.

Both the CLI and web endpoints now work if you set resourceType to all.

e.g. `conduit stat all`
2018-05-11 14:35:37 -07:00
Kevin Lingerfelt 4e8e1eb84d
CLI: Fix validation for service stats (#935)
* CLI: Fix validation for service stats
* Address review feedback

Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
2018-05-11 10:28:49 -07:00
Risha Mars f94856e489
Modify the Stat endpoint to also return the number of failed conduit pods (#895)
* Modify the Stat endpoint to also return the count of failed pods
* Add comments explaining pod count stats
* Rename total pod count to running pod count

This is to support the service mesh overview page, as I'd like to include an indicator of
failed pods there.
2018-05-08 10:35:21 -07:00
Andrew Seigner dce31b888f
Deprecate Tap, rename TapByResource to Tap (#844)
The `conduit tap` command is now deprecated.

Replace `conduit tap` with `connduit tapByResource`. Rename tapByResource
to tap. The underlying protobuf for tap remains, the tap gRPC endpoint now
returns Unimplemented.

Fixes #804

Signed-off-by: Andrew Seigner <siggy@buoyant.io>
2018-04-25 12:24:46 -07:00
Andrew Seigner a0a9a42e23
Implement Public API and Tap on top of Lister (#835)
public-api and and tap were both using their own implementations of
the Kubernetes Informer/Lister APIs.

This change factors out all Informer/Lister usage into the Lister
module. This also introduces a new `Lister.GetObjects` method.

Signed-off-by: Andrew Seigner <siggy@buoyant.io>
2018-04-24 18:10:48 -07:00
Andrew Seigner baf4ea1a5a
Implement TapByResource in Tap Service (#827)
The TapByResource endpoint was previously a stub.

Implement end-to-end tapByResource functionality, with support for
specifying any kubernetes resource(s) as target and destination.

Fixes #803, #49

Signed-off-by: Andrew Seigner <siggy@buoyant.io>
2018-04-23 16:13:26 -07:00
Andrew Seigner 79bdc638b3
Service support in stat command (#809)
The `stat` command did not support `service` as a resource type.

This change adds `service` support to the `stat` command. Specifically:
- as a destination resource on `--to` commands
- as a target resource on `--from` commands

Fixes #805

Signed-off-by: Andrew Seigner <siggy@buoyant.io>
2018-04-19 16:51:20 -07:00
Andrew Seigner 293e00bc3e
Introduce tapByResource cli command (#802)
The existing `tap` command is being deprecated.

Introduce a `tapByResource` cli command. It supports tapping a Kubernetes
resource or collection of resources, optionally filtered by outbound resources.
This command will eventually replace `tap`.

Part of #778

Signed-off-by: Andrew Seigner <siggy@buoyant.io>
2018-04-19 14:44:23 -07:00
Kevin Lingerfelt 653dc6bfaa
Add replication controller stats in CLI (#794)
* Add replication controller stats in CLI
* Fix pod status in stat summary tests

Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
2018-04-18 18:12:14 -07:00
Oliver Gould 06dd8d90ee
Introduce the TapByResource API (#778)
This changes the public api to have a new rpc type, `TapByResource`.
This api supersedes the Tap api. `TapByResource` is richer, more closely 
reflecting the proxy's capabilities.

The proxy's Tap api is extended to select over destination labels,
corresponding with those returned by the Destination api.

Now both `Tap` and `TapByResource`'s responses may include destination
labels.

This change avoids breaking backwards compatibility by:

* introducing the new `TapByResource` rpc type, opting not to change Tap
* extending the proxy's Match type with a new, optional, `destination_label` field.
* `TapEvent` is extended with a new, optional, `destination_meta`.
2018-04-18 15:37:07 -07:00
Kevin Lingerfelt 71a51afb40
Expose pod stats in CLI, web UI, and Grafana (#788)
* Expose pod stats in CLI, web UI, and Grafana
* Fix js api helpers test
* Add outbound traffic stats to pod dashboard

Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
2018-04-18 11:26:47 -07:00
Andrew Seigner 727521f914
Permit arbitrary time windows in public-api (#774)
The public-api previously only permitted 4 hard-coded time windows:
10s, 1m, 10m, 1h. This was primarily a relic of the recently removed
telemetry system.

Modify the public-api to validate the time string, but allow for any
window size, which is then passed through to Prometheus.

Fixes #686

Signed-off-by: Andrew Seigner <siggy@buoyant.io>
2018-04-16 17:37:17 -07:00
Kevin Lingerfelt 11a4359e9a
Misc cleanup following the telemetry rewrite (#771)
Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
2018-04-16 15:51:07 -07:00
Andrew Seigner 77fb6d3709
Add namespace as a resource type in public-api (#760)
* Add namespace as a resource type in public-api

The cli and public-api only supported deployments as a resource type.

This change adds support for namespace as a resource type in the cli and
public-api. This also change includes:
- cli statsummary now prints `-`'s when objects are not in the mesh
- cli statsummary prints `No resources found.` when applicable
- removed `out-` from cli statsummary flags, and analagous proto changes
- switched public-api to use native prometheus label types
- misc error handling and logging fixes

Part of #627

Signed-off-by: Andrew Seigner <siggy@buoyant.io>

* Refactor filter and groupby label formulation

Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>

* Rename stat_summary.go to stat.go in cli

Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>

* Update rbac privileges for namespace stats

Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
2018-04-13 16:53:01 -07:00
Andrew Seigner 21886760c6
Use apps/v1beta2 for Kubernetes 1.8 compatibility (#762)
Conduit was relying on apps/v1 to Deployment and ReplicaSet APIs.
apps/v1 is not available on Kubernetes 1.8. This prevented the
public-api from starting.

Switch Conduit to use apps/v1beta2. Also increase the Kubernetes API
cache sync timeout from 10 to 60 seconds, as it was taking 11 seconds on
a test cluster.

Fixes #761

Signed-off-by: Andrew Seigner <siggy@buoyant.io>
2018-04-13 12:08:16 -07:00
Kevin Lingerfelt fb15fe7c1a
Remove the telemetry service (#757)
* Remove the telemetry service

The telemetry service is no longer needed, now that prometheus scrapes
metrics directly from proxies, and the public-api talks directly to
prometheus. In this branch I'm removing the service itself as well as
all of the telemetry protobuf, and updating the conduit install command
to no longer install the service. I'm also removing the old version of
the stat command, which required the telemetry service, and renaming the
statsummary command to stat.

* Fix time window tests

* Remove deprecated controller scrape config

Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
2018-04-13 11:21:29 -07:00
Andrew Seigner e9b209829d
Handle NaN metrics (#750)
The Prometheus client sometimes returns NaN if a calculation is invalid,
such as histogram_quantile when no requests have occurred.

Add IsNaN check in the public-api and set output to zero.

Fixes #747

Signed-off-by: Andrew Seigner <siggy@buoyant.io>
2018-04-12 15:21:00 -07:00
Andrew Seigner 624b87f743
Implement ListPods in public-api (#743)
The ListPods endpoint's logic resides in the telemetry service, which is
going away.

Move ListPods logic into public-api, use new k8s informer APIs.

Fixes #694

Signed-off-by: Andrew Seigner <siggy@buoyant.io>
2018-04-11 17:53:57 -07:00
Kevin Lingerfelt 47caf1ca07
Add --all-namespaces flag to CLI statsummary command (#745)
* Add --all-namespaces flag to CLI statsummary command

* Fix statsummary output formatting

Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
2018-04-11 16:40:25 -07:00
Andrew Seigner 259fdcd134
Add latency stats in new stat summary endpoint (#737)
The new StatSummary endpoint was only providing request volume and
successs rate information.

Add support for retrieving latency stats via StatSummary. Also make
all prometheus calls in parallel, and implement kubernetes test
fixtures.

Fixes #681

Signed-off-by: Andrew Seigner <siggy@buoyant.io>
2018-04-11 11:58:32 -07:00
Kevin Lingerfelt 91c359e612
Switch public API to use cached k8s resources (#724)
* Switch public API to use cached k8s resources
* Move shared informer code to separate goroutine
* Fix spelling issue

Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
2018-04-10 11:39:31 -07:00
Andrew Seigner 3a341abe9a
Fix success rate calculation in public api (#723)
The success rate calculation relies on the `classification` label, but
was incorrectly specifying `fail` rather than `failure`.

Fix public api to specify `failure`. Also re-org public api tests for
easier Kubernetes and Prometheus mocking.

Signed-off-by: Andrew Seigner <siggy@buoyant.io>
2018-04-10 11:04:04 -07:00
Andrew Seigner 716b392231
Move StatSummary logic into grpc server (#717)
The StatSummary logic was implemented as a method on http_server.

Move the StatSummary logic into grpc_server, for consistency with the
other endpoints.

Signed-off-by: Andrew Seigner <siggy@buoyant.io>
2018-04-06 16:46:15 -07:00
Andrew Seigner 50c323c617
Use canonical k8s names, fix prom labels (#702)
The new statsummary command accepted friendly k8s names, which worked
for k8s queries, but Prometheus requires a specific key.

Modify the statsummary query to map friendly k8s names to canonical k8s
names when constructing the query. Then during the query, map the
canonical k8s name to a specific Prometheus label.

Fixes #695

Signed-off-by: Andrew Seigner <siggy@buoyant.io>
2018-04-06 12:34:54 -07:00
Risha Mars 2f5b5ea5f2
Start implementing conduit stat summary endpoint (#671)
Start implementing new conduit stat summary endpoint. 
Changes the public-api to call prometheus directly instead of the
telemetry service. Wired through to `api/stat` on the web server,
as well as `conduit statsummary` on the CLI. Works for deployments only.

Current implementation just retrieves requests and mesh/total pod count 
(so latency stats are always 0). 

Uses API defined in #663
Example queries the stat endpoint will eventually satisfy in #627

This branch includes commits from @klingerf 

* run ./bin/dep ensure
* run ./bin/update-go-deps-shas
2018-04-05 17:05:06 -07:00
Risha Mars d1a39ea6bf
Define a new telemetry Stat API (#663)
* Define a new telemetry Stat API

Proposal definition for a new Stat API, for the purposes of satisfying the queries proposed in #627.
StatSummary will replace Stat once implemented and the original Stat deleted.
2018-04-03 14:45:58 -07:00
Dennis Adjei-Baah 5a4c5aa683
Exclude telemetry generated by the control plane when requesting depl… (#493)
When the conduit proxy is injected into the controller pod, we observe controller pod proxy stats show up as an "outbound" deployment for an unrelated upstream deployment. This may cause confusion when monitoring deployments in the service mesh.

This PR filters out this "misleading" stat in the public api whenever the dashboard requests metric information for a specific deployment.

* exclude telemetry generated by the control plane when requesting deployment metrics

fixes #370

Signed-off-by: Dennis Adjei-Baah <dennis@buoyant.io>
2018-03-05 17:58:08 -08:00
Kevin Lingerfelt e57e74056e
Run go fix to fix context package imports (#470)
Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
2018-02-28 13:25:33 -08:00
Andrew Seigner 1db7d2a2fb
Ensure latency quantile queries match timestamps (#348)
In PR #298 we moved time window parsing (10s => (time.now - 10s,
time.now) down the stack to immediately before the query. This had the
unintended effect of creating parallel latency quantile requests with
slightly different timestamps.

This change parses the time window prior to latency quantile fan out,
ensuring all requests have the same timestamp.

Signed-off-by: Andrew Seigner <siggy@buoyant.io>
2018-02-13 16:26:54 -08:00
Andrew Seigner 50f4aa57e5
Require timestamp on all telemetry requests (#342)
PR #298 moved summary (non-timeseries) requests to Prometheus' Query
endpoint, with no timestamp provided. This Query endpoint returns a
single data point with whatever timestamp was provided in the request.
In the absense of a timestamp, it uses current server time. This causes
the Public API to return discreet data points with slightly different
timestamps, which is unexpected behavior.

Modify the Public API -> Telemetry -> Prometheus request path to always
require a timestamp for single data point requests.

Fixes #340

Signed-off-by: Andrew Seigner <siggy@buoyant.io>
2018-02-13 13:52:21 -08:00
Andrew Seigner 261586b862
Fix pointer copying (#330)
The Public APIs stat endpoint copies a slice of values to a slice of
pointers prior to gRPC response. Go's range clause re-uses the same
pointer for each iteration of the loop, causing a slice of {1,2,3}
becoming {3,3,3}.

Fix the range loop to directly reference pointers in the slice of
values, ignoring the range variable. Also add tests to catch this case.

Signed-off-by: Andrew Seigner <siggy@buoyant.io>
2018-02-10 11:04:28 -08:00
Andrew Seigner bffa5ff3e6
Concurrent Telemetry requests (#323)
All requests from the public API service to the Telemetry service were
done serially. In some cases a single request to the public API's Stat
endpoint resulted in 5 serial requests to the Telemetry service.

Make all requests from the Public API to Telemetry concurrent.

Signed-off-by: Andrew Seigner <siggy@buoyant.io>

Part of #299
2018-02-09 17:11:20 -08:00
Eliza Weisman 458e9d2ac5
Remove per-path metrics from telemetry pipeline (#317)
Follow-up from #315.

Now that the UIs don't report per-path metrics, we can remove the path label from Prometheus, the path aggregation and filtering options from the telemetry API, and the path field from the proxy report API.

I've modified the tests to no longer expect the removed fields, and manually verified that Conduit still works after making these changes.

Closes #265 

Signed-off-by: Eliza Weisman <eliza@buoyant.io>
2018-02-09 14:20:28 -08:00
Andrew Seigner 33e3c3ace9
Optimize Prometheus queries (#298)
Prometheus queries from the Telemetry service were taking seconds or 10s
of seconds.

Optimize these queries:
- Move all summary queries requiring a single point data off of Prometheus'
  QueryRange() endpoint, onto Query()
- Set `defaultVectorRange` to 30s, and also use it regardless of time
  window
Also add tests for grpc_server and telemetry server

Signed-off-by: Andrew Seigner <siggy@buoyant.io>

Fixes #260
2018-02-09 10:55:07 -08:00
Eliza Weisman 2015d992cc
Remove pod-level metrics from web and CLI (#304)
This PR updates the web UI to remove the pod detail page, and to remove the links to that page from pod names in metrics tables. It also removes the `pods` option from `conduit stat`, and the `sourcePod` and `targetPod` fields from the controller API proto's `MetricMetadata` message.

I've updated the `conduit stat` tests to reflect these changes, and manually verified the web UI changes.

Closes #261 

Signed-off-by: Eliza Weisman <eliza@buoyant.io>
2018-02-08 19:07:10 -08:00
Phil Calçado 9c03764a29
Remove hardcoded port and shared state for http test (#282)
We now create a new test HTTP server per test case instead of sharing it across them all.

This should solve the data races we have experienced on Travis.

Signed-off-by: Phil Calcado <phil@buoyant.io>
2018-02-06 13:48:14 -05:00
Andrew Seigner 4156af786d
Enable race detection in ci (#259)
We previously did not have race detection enabled because our tests
would fail. Following #249, this is no longer the case.

Enable race detection in ci and build instructions. This change also
fixes client_test.go attempting to allocate a 2GB buffer due to bad test
input.

Fixes #173

Signed-off-by: Andrew Seigner <siggy@buoyant.io>
2018-02-02 15:04:52 -08:00
Andrew Seigner 277c06cf1e
Simplify and refactor k8s labels and annnotations (#227)
The conduit.io/* k8s labels and annotations we're redundant in some
cases, and not flexible enough in others.

This change modifies the labels in the following ways:
`conduit.io/plane: control` => `conduit.io/controller-component: web`
`conduit.io/controller: conduit` => `conduit.io/controller-ns: conduit`
`conduit.io/plane: data` => (remove, redundant with `conduit.io/controller-ns`)
It also centralizes all k8s labels and annotations into
pkg/k8s/labels.go, and adds tests for the install command.

Part of #201

Signed-off-by: Andrew Seigner <siggy@buoyant.io>
2018-02-01 14:12:06 -08:00
Risha Mars a9d4a3d74e
Add more prometheus instrumentation (latency, response size) (#174)
We added basic prometheus instrumentation, but this only encapsulated basic go metrics and
 request counts. This adds latency and response size metrics exporting as well, to the 
public-api server, theweb server and the telemetry server.

Since the util function in grpc.go was basically used to wrap the server creation in a prometheus handler, I added the other prometheus constants in there and renamed the file to prometheus.go. 

- Add request duration and response size instrumentation to web and public api
- Also add latency monitoring to telemetry service requests
- Rename util/grpc.go to util/prometheus.go
2018-02-01 09:50:31 -08:00
Kevin Lingerfelt 4a76c6448b
Update cli subcommands to print errors when encountered (#221)
Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
2018-01-29 11:28:19 -08:00
Kevin Lingerfelt 7399df83f1
Set conduit version to match conduit docker tags (#208)
* Set conduit version to match conduit docker tags

Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>

* Remove --skip-inbound-ports for emojivoto

Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>

* Rename git_sha => git_sha_head

Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>

* Switch to using the go linker for setting the version

Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>

* Log conduit version when go servers start

Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>

* Cleanup conduit script

Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>

* Add --short flag to head sha command

Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>

* Set CONDUIT_VERSION in docker-compose env

Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
2018-01-26 11:43:45 -08:00
Phil Calçado 9410da471a
Better error handling for Tap (#177)
Previously, running `$conduit tap` would return a `Unexpected EOF` error when the server wasn't available. This was due to a few problems with the way we were handling errors all the way down the tap server. This change fixes that and cleans some of the protobuf-over-HTTP code.

- first step towards #49
- closes #106
2018-01-25 11:49:38 -05:00
Dennis Adjei-Baah f7af375e73
Remove scheme requirement for api-addr flag in conduit CLI (#126)
* Allow external controller public api clients that don't rely on a kubeconfig to interact with Conduit CLI

Signed-off-by: Dennis Adjei-Baah <dennis@buoyant.io>
2018-01-17 17:12:44 -08:00
Kevin Lingerfelt fd3cfcb5d9
Move healthcheck proto to separate file, use throughout (#150)
* Move healthcheck proto to separate file, use throughout

Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>

* Remove Check message from healthcheck.proto

Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>

* Standardize healthcheck protobuf import name

Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
2018-01-17 11:15:38 -08:00
Phil Calçado 612bd0f7a0
Add --verbose option to CLI (#154)
* Use stdout as writer for tap command

fixes #136

Signed-off-by: Phil Calcado <phil@buoyant.io>

* Add --log-level to command line

Signed-off-by: Phil Calcado <phil@buoyant.io>
2018-01-17 12:06:43 -05:00
Phil Calçado e328db7e87
Adds conduit-api check for status command (#140)
* Abstract Conduit API client from protobuf interface to add new features

Signed-off-by: Phil Calcado <phil@buoyant.io>

* Consolidate mock api clients

Signed-off-by: Phil Calcado <phil@buoyant.io>

* Add simple implementation of healthcheck for conduit api

Signed-off-by: Phil Calcado <phil@buoyant.io>

* Change NextSteps to FriendlyMessageToUser

Signed-off-by: Phil Calcado <phil@buoyant.io>

* Add grpc check for status on the client

Signed-off-by: Phil Calcado <phil@buoyant.io>

* Add simple server-side check for Conduit API

Signed-off-by: Phil Calcado <phil@buoyant.io>

* Fix feedback from PR

Signed-off-by: Phil Calcado <phil@buoyant.io>
2018-01-12 15:35:22 -05:00
Phil Calçado 709de5a7b0
Moves k8s and conduit client code to /pkg (#103)
* Rename constructor functions from MakeXyz to NewXyz

As it is more commonly used in the codebase

Signed-off-by: Phil Calcado <phil@buoyant.io>

* Make Conduit client depend on KubernetesAPI

Signed-off-by: Phil Calcado <phil@buoyant.io>

* Move Conduit client and k8s logic to standard go package dir for internal libs

Signed-off-by: Phil Calcado <phil@buoyant.io>

* Move dependencies to /pkg

Signed-off-by: Phil Calcado <phil@buoyant.io>

* Make conduit client more testable

Signed-off-by: Phil Calcado <phil@buoyant.io>

* Remove unused config object

Signed-off-by: Phil Calcado <phil@buoyant.io>

* Add more test cases for marhsalling

Signed-off-by: Phil Calcado <phil@buoyant.io>

* Move client back to controller

Signed-off-by: Phil Calcado <phil@buoyant.io>

* Sort imports

Signed-off-by: Phil Calcado <phil@buoyant.io>
2018-01-04 10:10:10 -08:00
Kevin Lingerfelt a8e75115ab
Prepare the repo for the v0.1.1 release (#75)
* Prepare the repo for the v0.1.1 release

* Add changelog

* Changelog updates, wrap at 100 characters
2017-12-20 10:51:53 -08:00
Phil Calçado 0a6a9edaee
Respect $KUBECONFIG env var (#68)
* Move kubectl logis to k8s package

* Made kubectl return *url.URL, just like API

* Make k8s API code respect /Users/pcalcado/.kube/config (closes #17)

* Fix style mistakes and typos
2017-12-20 11:50:25 +11:00
Kevin Lingerfelt 2f114e69fa
Add support for path stats in cli and web api (#13)
* Add support for path stats in cli and web api

The cli stat command supports grouping by pod and deployment. With this
change, it will also support grouping by path, in order to facilitate a
summary stats per individual endpoint.

* Right-align numeric columns in stat output
2017-12-08 12:24:39 -08:00
Kevin Lingerfelt 906d4e8b69
Fix public-api error marshaling and unmarshaling (#16) 2017-12-08 11:03:55 -08:00
Oliver Gould bff3efea3f
Prepare for v0.1.0 (#1)
Update versions in code.

Use default docker tag of v0.1.0
2017-12-04 19:55:56 -08:00
Oliver Gould b104bd0676 Introducing Conduit, the ultralight service mesh
We’ve built Conduit from the ground up to be the fastest, lightest,
simplest, and most secure service mesh in the world. It features an
incredibly fast and safe data plane written in Rust, a simple yet
powerful control plane written in Go, and a design that’s focused on
performance, security, and usability. Most importantly, Conduit
incorporates the many lessons we’ve learned from over 18 months of
production service mesh experience with Linkerd.

This repository contains a few tightly-related components:
- `proxy` -- an HTTP/2 proxy written in Rust;
- `controller` -- a control plane written in Go with gRPC;
- `web` -- a UI written in React, served by Go.
2017-12-05 00:24:55 +00:00