* Add namespace as a resource type in public-api
The cli and public-api only supported deployments as a resource type.
This change adds support for namespace as a resource type in the cli and
public-api. This also change includes:
- cli statsummary now prints `-`'s when objects are not in the mesh
- cli statsummary prints `No resources found.` when applicable
- removed `out-` from cli statsummary flags, and analagous proto changes
- switched public-api to use native prometheus label types
- misc error handling and logging fixes
Part of #627
Signed-off-by: Andrew Seigner <siggy@buoyant.io>
* Refactor filter and groupby label formulation
Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
* Rename stat_summary.go to stat.go in cli
Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
* Update rbac privileges for namespace stats
Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
Conduit was relying on apps/v1 to Deployment and ReplicaSet APIs.
apps/v1 is not available on Kubernetes 1.8. This prevented the
public-api from starting.
Switch Conduit to use apps/v1beta2. Also increase the Kubernetes API
cache sync timeout from 10 to 60 seconds, as it was taking 11 seconds on
a test cluster.
Fixes#761
Signed-off-by: Andrew Seigner <siggy@buoyant.io>
* Remove the telemetry service
The telemetry service is no longer needed, now that prometheus scrapes
metrics directly from proxies, and the public-api talks directly to
prometheus. In this branch I'm removing the service itself as well as
all of the telemetry protobuf, and updating the conduit install command
to no longer install the service. I'm also removing the old version of
the stat command, which required the telemetry service, and renaming the
statsummary command to stat.
* Fix time window tests
* Remove deprecated controller scrape config
Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
The new StatSummary endpoint was only providing request volume and
successs rate information.
Add support for retrieving latency stats via StatSummary. Also make
all prometheus calls in parallel, and implement kubernetes test
fixtures.
Fixes#681
Signed-off-by: Andrew Seigner <siggy@buoyant.io>
* Switch public API to use cached k8s resources
* Move shared informer code to separate goroutine
* Fix spelling issue
Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
The Grafana dashboards key off of deployment, but had no awareness of
namespaces, causing incorrect metrics aggregation and display.
This change makes the Grafana dashboards key off of namespaces, and also
modifies the Grafana links in the Conduit dashboard to link to
namespace+deployment.
Fixes#704
Part of #420
Signed-off-by: Andrew Seigner <siggy@buoyant.io>
The new statsummary command accepted friendly k8s names, which worked
for k8s queries, but Prometheus requires a specific key.
Modify the statsummary query to map friendly k8s names to canonical k8s
names when constructing the query. Then during the query, map the
canonical k8s name to a specific Prometheus label.
Fixes#695
Signed-off-by: Andrew Seigner <siggy@buoyant.io>
* Link to Grafana from Conduit Dashboard
Previously the only way to access the Grafana dashboards was via direct
link, provided by the `conduit dashboard` command.
Add Grafana links throughout the Conduit Dashboard, next to all
Deployment objects. This change also modifies the behavior of the
ConduitLink helper, to enable linking to other deployments proxied by
the `conduit dashboard` command.
Part of #420
Signed-off-by: Andrew Seigner <siggy@buoyant.io>
* review feedback
Signed-off-by: Andrew Seigner <siggy@buoyant.io>
* review feedback, fix console, remove absolute
Signed-off-by: Andrew Seigner <siggy@buoyant.io>
Start implementing new conduit stat summary endpoint.
Changes the public-api to call prometheus directly instead of the
telemetry service. Wired through to `api/stat` on the web server,
as well as `conduit statsummary` on the CLI. Works for deployments only.
Current implementation just retrieves requests and mesh/total pod count
(so latency stats are always 0).
Uses API defined in #663
Example queries the stat endpoint will eventually satisfy in #627
This branch includes commits from @klingerf
* run ./bin/dep ensure
* run ./bin/update-go-deps-shas
* fix pod status and count display in control plane dashboard section:
- the control plane would show terminated and stale deployments in the UI, this is confusing and might indicate errors
- this filters out temrinated and failed component deploys from the UI
- it is to note that pending deploys will still be counted and represented with a greyed out status dot
- Fixes: #606
Signed-off-by: Franziska von der Goltz <franziska@vdgoltz.eu>
Signed-off-by: Franziska von der Goltz <franziska@vdgoltz.eu>
remove toggle sorting functionality from TableComponent:
- tables displaying metrics allowed to toggle between being sorted and unsorted when clicking the same button. This was confusing behavior for the user.
- this PR removes the toggle functionality and introduces a BaseTable Component that extends antd's component without the capability to toggle
- Fixes: #566
Signed-off-by: Franziska von der Goltz <franziska@vdgoltz.eu>
* Add tests/utils/scripts for running integration tests
Add a suite of integration tests in the `test/` directory, as well as
utilities for testing in the `testutil/` directory.
You can use the `bin/test-run` script to run the full suite of tests,
and the `bin/test-cleanup` script to cleanup after the tests.
The test/README.md file has more information about running tests.
@pcalcado, @franziskagoltz, and @rmars also contributed to this change.
* Create TEST.md file at the root of the repo
* Update based on review feedback
* Relax external service IP timeout for GKE
* Update TEST.md with more info about different types of test runs
* More updates to TEST.md based on review feedback
Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
Shortly after conduit is installed in k8s environment. The control plane component that establishes a watch endpoint with k8s run in to networking issues during proxy initialization. During failure, each watcher fails to retry its connection to k8s watch endpoint which leads to timeouts and eventually, multiple controller pod restarts.
This PR adds retry logic to each "watch" enabled package.
fixes#478
Signed-off-by: Dennis Adjei-Baah <dennis@buoyant.io>
* Use Go 1.10.0 to build Go components.
Take advantage of the new build cache in Go 1.10. Future work on improving
build performance will utilize the build cache further.
Signed-off-by: Brian Smith <brian@briansmith.org>
Previously Dockerfile-go-deps was converted from a multi-stage Dockefile
to a single-stage Dockerfile in anticipation of enabling efficient use
of `--cache-from` in CI. However, that resulted in the image ballooning
in size because it contained the Git repo for every package downloaded
by `dep ensure`.
Bring the image back down to the proper size by removing the temporary
files created.
Signed-off-by: Brian Smith <brian@briansmith.org>
* Display more decimal points for truncated numbers, add hover info
* Filter completed pods out of web UI
* Decrease the polling interval from 10s to 2s
* Add more detailed pod categorization based on status
* Tweak filtering of pods, tweak explanations in status table
- reduce row spacing on tables to make them more compact
- Rename TabbedMetricsTable to MetricsTable since it's not tabbed any more
- Format latencies greater than 1000ms as seconds
- Make sidebar collapsible
- poll the /pods endpoint from the sidebar in order to refresh the list of deployments in the autocomplete
- display the conduit namespace in the service mesh details table
- Use floats rather than Col for more responsive layout (fixes#224)
On my system (i9-7960x running Docker natively in Linux) this regularly saves
over 11 seconds of build time when a file under pkg/ changes and over 1.5
seconds of build time when a file under controller/ changes. Since most
contributors are running Docker in a VM on less powerful computers, the
savings for most contributors should be significantly greater.
I imagine the savings for web/ and cli/ and proxy-init/ are similar, but I
did not measure them.
Signed-off-by: Brian Smith <brian@briansmith.org>
Previously Dockerfile-go-deps would run `dep ensure` whenever anything in the
source tree changed. Also, because it was a multi-stage Dockerfile it did not
work well with Docker's `--cache-from` feature.
Change Dockerfile-go-deps to only re-run `dep ensure` when Gopkg.{toml,lock}
and/or bin/dep change. Simplify it to a single stage so that it works better
with Docker's `--cache-from` feature.
Signed-off-by: Brian Smith <brian@briansmith.org>
bin/dep verifies the digest of the `dep` downloaded `dep` executable,
whereas previously Dockerfile-go-deps wasn't.
Signed-off-by: Brian Smith <brian@briansmith.org>
UI cleanups. Remove repetitive labels in the UI, remove unused elements,
remove graphs until we improve their utility.
- remove “Deployment” from the headers of the Deployment Detail Page
- remove Routes in sidebar
- kill leftmost 100px of sidebear
- remove word controller from service mesh page first table
- add twitter and GitHub and slack links
- kill the graphs, replace with one large header (request rate, success rate, latency top bar)
put upstream/downstream diagram before upstream downstream tables
* Clean up DeploymentList page (#321)
- remove "Most active deployments" graphs from the Deployments List page
- remove the scatterplot sections of the page as I don't think we'll be using them for a while
Follow-up from #315.
Now that the UIs don't report per-path metrics, we can remove the path label from Prometheus, the path aggregation and filtering options from the telemetry API, and the path field from the proxy report API.
I've modified the tests to no longer expect the removed fields, and manually verified that Conduit still works after making these changes.
Closes#265
Signed-off-by: Eliza Weisman <eliza@buoyant.io>
I've removed per-path metrics from the web dashboard and from the `conduit stat` command.
Manually validated that these metrics are no longer displayed.
Closes #263
Signed-off-by: Eliza Weisman <eliza@buoyant.io>
Prometheus queries from the Telemetry service were taking seconds or 10s
of seconds.
Optimize these queries:
- Move all summary queries requiring a single point data off of Prometheus'
QueryRange() endpoint, onto Query()
- Set `defaultVectorRange` to 30s, and also use it regardless of time
window
Also add tests for grpc_server and telemetry server
Signed-off-by: Andrew Seigner <siggy@buoyant.io>
Fixes#260
This PR updates the web UI to remove the pod detail page, and to remove the links to that page from pod names in metrics tables. It also removes the `pods` option from `conduit stat`, and the `sourcePod` and `targetPod` fields from the controller API proto's `MetricMetadata` message.
I've updated the `conduit stat` tests to reflect these changes, and manually verified the web UI changes.
Closes#261
Signed-off-by: Eliza Weisman <eliza@buoyant.io>
* Consolidate latency, success, request metrics into one tab
on the SortableMetricsTable
- removes sparklines from the table
- makes tables sortable by default
- move pod table in DeploymentDetail to its own row
* remove request distribution column, reorder columns
* Control metricsWindow from root of app
- Add buttons [currently hidden] on metrics pages to control window of metrics requests
- Consolidate metricsWindow usage (stop passing it around)
- Add a ConduitLink component so we can stop passing around pathPrefix
- Add tests for ApiHelpers
* Hide the time window buttons; fix bug in absolute links
* Add a note explaining why metricWindow buttons are disabled
* Convert ConduitLink in to a component that wraps another
The conduit dashboard command asychronously shells out and runs "kubectl
proxy".
This change replaces the shelling out with calls to kubernetes proxy
APIs. It also allows us to enable race detection in our go tests, as the
shell out code tests did not pass race detection.
Fixes#173
Signed-off-by: Andrew Seigner <siggy@buoyant.io>
* Add `bin/dep` which fetches a fixed version of `dep` to be used.
* Upgrade from dep 0.3.1 to 0.4.1
* Fix inconsistent Gopkg.lock by checking in the result of `bin/dep ensure`
Signed-off-by: Alex Leong <alex@buoyant.io>
The conduit.io/* k8s labels and annotations we're redundant in some
cases, and not flexible enough in others.
This change modifies the labels in the following ways:
`conduit.io/plane: control` => `conduit.io/controller-component: web`
`conduit.io/controller: conduit` => `conduit.io/controller-ns: conduit`
`conduit.io/plane: data` => (remove, redundant with `conduit.io/controller-ns`)
It also centralizes all k8s labels and annotations into
pkg/k8s/labels.go, and adds tests for the install command.
Part of #201
Signed-off-by: Andrew Seigner <siggy@buoyant.io>
We added basic prometheus instrumentation, but this only encapsulated basic go metrics and
request counts. This adds latency and response size metrics exporting as well, to the
public-api server, theweb server and the telemetry server.
Since the util function in grpc.go was basically used to wrap the server creation in a prometheus handler, I added the other prometheus constants in there and renamed the file to prometheus.go.
- Add request duration and response size instrumentation to web and public api
- Also add latency monitoring to telemetry service requests
- Rename util/grpc.go to util/prometheus.go
* Various small UI naming tweaks
- align top two tables in the service mesh page
- "All Deployments" -> "Deployments"
- reorder latency p50, p95, p99
- "Current success" -> "Success rate"
* Add margin to incomplete mesh message, reorder latency in TabbedMetricsTable
* Right align numbers in service mesh page
* Set conduit version to match conduit docker tags
Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
* Remove --skip-inbound-ports for emojivoto
Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
* Rename git_sha => git_sha_head
Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
* Switch to using the go linker for setting the version
Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
* Log conduit version when go servers start
Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
* Cleanup conduit script
Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
* Add --short flag to head sha command
Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
* Set CONDUIT_VERSION in docker-compose env
Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
* Switch to ant sider/content Layout modules, to help style sidebar
This fixes the problem of the sidebar not extending all the way on long pages.
* Fix a bug where the autocomplete options weren't being reset when an item was selected
* Rename components to clarify component relationships
* Rename Deployment to DeploymentDetail to match PodDetail
* Rename Deployments to DeploymentsList to clarify which page this is
* Rename StatPane to ResourceMetricsOverview to be a less generic name
* Rename HealthPane -> ResourceHealthOverview
* Rename StatPaneStat -> ResourceOverviewMetric
Signed-off-by: Risha Mars <mars@buoyant.io>
Previously, proxy-deps and go-deps included the source tree for local
projects. This can cause build conflicts when files are renamed.
By adopting a multi-stage build for the proxy-deps image, we can be sure
that we only preserve essential dependencies & manifests in the
proxy-deps and go-deps images.
Furthermore, `bin/update-go-deps-shas` and `bin/update-proxy-deps-shas` have
been added to ease maintenance when files are changed.
Fixes#159
Signed-off-by: Oliver Gould <ver@buoyant.io>
* Use more descriptive CSS variable names (#135)
Signed-off-by: Nathan Murthy <drip.dr0p.dr0p@gmail.com>
* Decouple the names of the colors from their use
Signed-off-by: Nathan Murthy <drip.dr0p.dr0p@gmail.com>
* Rename a few more root CSS color var names
Signed-off-by: Nathan Murthy <drip.dr0p.dr0p@gmail.com>
* Update more CSS color vars referenced elsewhere
Signed-off-by: Nathan Murthy <drip.dr0p.dr0p@gmail.com>
* Add autocomplete search bar back into sidebar
Signed-off-by: Risha Mars <mars@buoyant.io>
* Use getPodsByDeployment
Signed-off-by: Risha Mars <mars@buoyant.io>
* Revive scatterplot: re-add scatterplot to Deployments page
Tried to make some UI improvements to address previous problems:
* added a hover bar and tooltip that displays all of the nodes under the bar,
in descending order of successRate (to correspond with their order in the chart)
* the tooltip looked weird in the empty state so I also added the max/min latencies
observed there
Also cleans up the Deployments page a little when there are not any "least healthy deployments".
* Previously, the sidebar tooltip would still render the last
highlighted nodes' information when the dots updated. Fix that
by selecting a datapoint to highlight when the dots update.
* Add overlay tooltip with names of highlighted nodes
* Align the node labels with the node, except in cases of label overlap
Signed-off-by: Risha Mars <mars@buoyant.io>
* Use stdout as writer for tap command
fixes#136
Signed-off-by: Phil Calcado <phil@buoyant.io>
* Add --log-level to command line
Signed-off-by: Phil Calcado <phil@buoyant.io>
Our build instructions were scattered across a few README's.
This consolidates all instructions relevant to Conduit development into
a single BUILD.md.
Fixes#134
Signed-off-by: Andrew Seigner <andrew@sig.gy>
* Abstract Conduit API client from protobuf interface to add new features
Signed-off-by: Phil Calcado <phil@buoyant.io>
* Consolidate mock api clients
Signed-off-by: Phil Calcado <phil@buoyant.io>
* Add simple implementation of healthcheck for conduit api
Signed-off-by: Phil Calcado <phil@buoyant.io>
* Change NextSteps to FriendlyMessageToUser
Signed-off-by: Phil Calcado <phil@buoyant.io>
* Add grpc check for status on the client
Signed-off-by: Phil Calcado <phil@buoyant.io>
* Add simple server-side check for Conduit API
Signed-off-by: Phil Calcado <phil@buoyant.io>
* Fix feedback from PR
Signed-off-by: Phil Calcado <phil@buoyant.io>
* add ability to add flash to latest datapoint on line graph based on passed down prop.
- added flash to all graphs except the sparklines
Signed-off-by: Franziska von der Goltz <franziska@vdgoltz.eu>
* Add /paths page that shows rollup metrics by path
* Clean up ApiHelpers a bit
Adds ability to sort by column in the tabbed metrics table (to make a TabbedMetricsTable sortable, set sortable={true})
Adds a page, accessible via /paths that shows a table of all paths, with their request/success/latency metrics. I haven't exposed it in the sidebar as it doesn't have design treatment.
* Sort imports
Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
* Upgrade k8s.io/client-go to v6.0.0
Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
* Make k8s store initialization blocking with timeout
Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
* Upgrade ant to 3.1.0
* Adjust styles for updated ant
- Locale is enUS by default now, so removing our config
- Adjust table styles
* Upgrade react and react-dom to 16.2.0
* Upgrade enzyme to 3.3.0, fix tests accordingly
The existing startup/shutdown log info messages had spacing issues and
used fmt.
Update the log messages to use logrus for consistency, and fix spacing
issues.
Signed-off-by: Andrew Seigner <andrew@sig.gy>
The image tags for gcr.io/runconduit/go-deps and
gcr.io/runconduit/proxy-deps were not updating to account for all
changes in those images.
Modify SHA generation to include all files that affect the base
dependency images. Also add instructions to README.md for updating
hard-coded SHAs in Dockerfile's.
Fixes#115
Signed-off-by: Andrew Seigner <andrew@sig.gy>
The web README.md mentions running locally, but it's not explicit about
how to connect to public-api.
Add notes on various ways to connect a local web to a public-api
process.
Signed-off-by: Andrew Seigner <andrew@sig.gy>
* Rename constructor functions from MakeXyz to NewXyz
As it is more commonly used in the codebase
Signed-off-by: Phil Calcado <phil@buoyant.io>
* Make Conduit client depend on KubernetesAPI
Signed-off-by: Phil Calcado <phil@buoyant.io>
* Move Conduit client and k8s logic to standard go package dir for internal libs
Signed-off-by: Phil Calcado <phil@buoyant.io>
* Move dependencies to /pkg
Signed-off-by: Phil Calcado <phil@buoyant.io>
* Make conduit client more testable
Signed-off-by: Phil Calcado <phil@buoyant.io>
* Remove unused config object
Signed-off-by: Phil Calcado <phil@buoyant.io>
* Add more test cases for marhsalling
Signed-off-by: Phil Calcado <phil@buoyant.io>
* Move client back to controller
Signed-off-by: Phil Calcado <phil@buoyant.io>
* Sort imports
Signed-off-by: Phil Calcado <phil@buoyant.io>
* Make TabbedMetricsTable in charge of fetching timeseries
All of the parent components in charge of fetching timeseries data
don't actually use them, but pass them to this table. It would simplify
a lot of the parents if this component handled the ts fetching too.
This introduces a urlsForResource helper in ApiHelpers to allow
us to consolidate url generating in the app to one place. It also
associates the appropriate groupBy for various fetches.
Signed-off-by: Risha Mars <mars@buoyant.io>
Previously Conduit would render an iframe, received from
versioncheck.conduit.io.
Modify the client to retrieve the latest released version, via CORS.
Signed-off-by: Andrew Seigner <andrew@sig.gy>
The README.md in the Web project describes building and running the Web
service locally, against a public API server in Docker.
Add instructions for running the Web Service locally against a public
API Server in Kubernetes.
Signed-off-by: Andrew Seigner <andrew@sig.gy>
* Have better empty states in Deployment Detail and Pod Detail
* Fix call to action on deployment detail page
Adds various checks to hide sections of pages that don't have data:
* Removes the timeseries graphs and latency overview from Deployment/Pod
detail if there are no metrics for it
* Removes the Upstreams/Downstreams if num upstreams / downstreams is 0
* Removes the pod barcharts from the Deployment detail if there are no pods
* If there are no pod metrics, populate the list of pods from /pods
These changes strip the pod detail view down a lot if there's no pod data, but that might be fine
* fix inconsistent deployment count on servicemesh page.
* tests added for deploy count messaging on servicemesh page
* refactored code for Call To Action component to use 'instructions' in util
* refactored correlating css
* Remove integration test clause from travis file
* Use correct channel for asyn process error reporting
* Fix test for sorting stat keys
* Make integration tests only run if CONDUIT_INTEGRATION_TESTS_ENABLED is set
* Fix timeouts in tests
* Fix handler test check for version
* Removes smoke test that required kubectl as not present on travis
* Replace tail with sleep to avoid leaking subprocesses during tests
* Fix typo & extract constant
* fix sidebar highlighting when dashboard is opened via cli:
- took path prefix into account
* addressed review feedback: using pathprefix
* addressed review feedback:
- revert to using this.props.location
* updates sidebar, initial progress indicator
* Update web README
* Revamp metrics processing in MetricUtils
- Try to improve documentation of functions
- Remove nested data from processed metrics - they were often duplicated data
- Split out timeseries processing from rollup processing
- Remove the 'rollup' level of nesting
- Limit the number of timeseries we fetch on the deployments page
Fix various loose ends in the web app:
* Add Tooltip to status dots explaining color meaning
* Adjust formatting of RPS numbers
* Pass in current success rate to the HealthPane
* Fix page header styling caused by content being in wrong div
* Align the metric value for inbound SR in Health Pane
* Status Dot tweaks
* Improve rendering of status dots when there are a lot of them
* Add support for path stats in cli and web api
The cli stat command supports grouping by pod and deployment. With this
change, it will also support grouping by path, in order to facilitate a
summary stats per individual endpoint.
* Right-align numeric columns in stat output
We’ve built Conduit from the ground up to be the fastest, lightest,
simplest, and most secure service mesh in the world. It features an
incredibly fast and safe data plane written in Rust, a simple yet
powerful control plane written in Go, and a design that’s focused on
performance, security, and usability. Most importantly, Conduit
incorporates the many lessons we’ve learned from over 18 months of
production service mesh experience with Linkerd.
This repository contains a few tightly-related components:
- `proxy` -- an HTTP/2 proxy written in Rust;
- `controller` -- a control plane written in Go with gRPC;
- `web` -- a UI written in React, served by Go.
Andrew Seigner
Kevin Lingerfelt
Risha Mars
Franziska von der Goltz
Brian Smith
Dennis Adjei-Baah
Eliza Weisman
Alex Leong
Oliver Gould
Nathan Murthy
Phil Calçado
Sterling White