Commit Graph

51 Commits

Author SHA1 Message Date
Dave Protasowski aff7376aba
drop stackdriver metrics and tracing exporters (#2183)
* drop stack driver as a tracing backend

* drop stackdriver as a metrics backend

* update deps - dropping stackdriver

* fix linting issues

* drop further references to stack driver

* drop serving & eventing metric key constants
these have been moved to their respective repos
see: https://github.com/knative/pkg/issues/608

* move source metrickeys to pkg/source

We're going to move the StatsReporter downstream eventually

related:
https://github.com/knative/eventing/pull/5587
https://github.com/knative/eventing/pull/5586

* fix linter
2021-07-14 11:45:31 -07:00
Markus Thömmes 40488532be
Drop master references to knative repos (#2050) 2021-03-11 09:48:26 -08:00
Matt Moore 27db5ac24c
Add nostackdriver to metrics as well (#1975) 2020-12-23 18:48:04 -08:00
Evan Anderson e41409af6c
Refactor and fix metrics export tests. (#1957)
* Copy resource_view_test to e2e_test

* Restore metrics/resource_view_test.go

* Refactor metrics tests, fix e2e_test

* Address yanweiguo feedback

* Cleanup per vagababov

* Switch to zap.Error but avoid the bug the code previously had.
2020-12-18 10:57:03 -08:00
Stavros Kontopoulos c73d344c89
Add host config option to the prometheus exporter (#1882)
* add host to the prometheus exporter

* typo

* fix imports

* updates

* fixes

* more fixes

* typos

* respect the default

* fix comment

* use default

* nits
2020-11-04 11:38:04 -08:00
Victor Agababov c4ce2ffa6a
Remove deprecated methods now that downstream has been updated (#1847) 2020-10-26 13:02:42 -07:00
Markus Thömmes 22e4e0a702
Fix API stutter in metric option functions (#1837) 2020-10-23 09:37:37 -07:00
Victor Agababov 324e49bc7f
Some cleanups arounds the metrics code (#1834)
- use switch
- add new method that won't trigger lint, with a deprecation notice when deps are updated
- other minor things
2020-10-22 18:21:37 -07:00
Markus Thömmes 385c8b9c0e
Fix nolint warnings and adhere to best practices (#1823) 2020-10-20 09:33:59 -07:00
Victor Agababov 7bad843466
Enable golint and exclude some other generated or additional dirs (#1783)
* Enable golint and exclude some other generated or additional dirs

Also remove `test` ignore, since it's covered by path ignore rule.

* meh

* fixes

* more

* progressing

* further

* like a boss
2020-10-07 14:58:20 -07:00
Markus Thömmes 5fbbde31b3
Align linters with serving (enables stylecheck and asciicheck) (#1738) 2020-09-23 07:37:40 -07:00
Markus Thömmes 754da4653a
Enable unparam linter to catch unused parameters (#1731) 2020-09-22 09:13:40 -07:00
Zbynek Roubalik 2d4efecc6b
bump to k8s 1.18 (#1428)
* bump to k8s 1.18.8

Signed-off-by: Zbynek Roubalik <zroubali@redhat.com>

* plumbing ctx through

Signed-off-by: Zbynek Roubalik <zroubali@redhat.com>

* add more ctx plumbing

Signed-off-by: Zbynek Roubalik <zroubali@redhat.com>

* ctx WithCancel()

Signed-off-by: Zbynek Roubalik <zroubali@redhat.com>
2020-09-11 07:54:00 -07:00
Markus Thömmes c903170f41
Completely skip metric reporting if backend is 'none'. (#1599) 2020-08-10 10:22:30 -07:00
jjzeng-seattle bac0302109
Set reporting period for all meters (#1575)
* Set reporting period for all meters

* using switch statement to simplify

* Change 60 seconds to a minute

* Add a test case for openCencus customed reporting period
2020-08-04 15:06:28 -07:00
Evan Anderson e7bd895fbd
Introduce `pkg/metrics` support for multiple Resources (#1392)
* Update OpenCensus to include #1196.

* Extend OpenCensus to track per-Resource stats.

* Add generic ResourceView mechanics, part 1. TODO: wire newFactory, tests

* Move factory creation to where it can be customized per-backend

* Integrate opencensus export with Resource-based exporters

* Add stackdriver support for resource views

* address feedbacks

* add test cases

* make KnativeRevision a map

* Update go modules to point to latest opencensus which includes `view.Meter`.

* Add support for Resources to Prometheus Exporter.

This involved copying/improving much of the contrib prometheus exporter.

* add resource_view tests and fix minor issues

* add recorder tests for meter

* Update resource_view_test with an e2e export test.

* Fix prometheus e2e resource_view_test.

Found a fun circular lock.

Found a fun double-registration in metricproducer/Meter.

Found some unnecessary extra code.

Fixed prometheus string sanitization.

Removed a layer of Prometheus wrapping which made debugging more challenging.

* Add test for standard view exports and non-resourced views to e2e test.

* Add UnregisterResourceView to fix tests

* Add OpenCensus e2e resource export test.

Still a little flaky, so work around that with "<=".

* Fix tests for OpenCensus export.

* Fix flakiness / timing issues in OpenCensus exporter e2e test.

* Update to latest ocagent to fix `Resources` bug.

* Don't swallow errors from stackdriver init.

* Get Stackdriver export mostly working in e2e tests

Tests still fail because stackdriver converts custom metrics to a namespaced format and seems to lose Resource information.

* Add support for Stackdriver resource export.

* ./hack/update-codegen.sh

* Fix data race with graceful shutdown.

* Fix issues noticed by @vagababov

* Address mattmoor-bot comments.

* Remove MonitoredResource and apply suggestions from yanweiguo

* More cleanup to stackdriver_exporter_test.go

* Update tests (particularly config_test.go) for Stackdriver recorder changes which sometimes record with additional resources.

* Avoid setting two `view.WithRecorder` options on record

* Add stackdriver tests

* Address comments

* Switch Prometheus port due to possible conflict in test env

* Fix missing :=

* FIx data races around *view.Data shared across Meters.

Cleanup resource_view_test.

* Remove unneeded monitored_resources.go

* Use sync.WaitGroup to avoid data races on closing channel.

* Sort the map for ResourceAsString

* Apply lint and vagababov changes.

* Fix typos

* refactor ResourceAsString

* Update stackdriver exporter version

* Update metrics/resource_view_test.go

Remove accidental paste

* Fix go.sum

* Address yanweiguo comments on comments

* Fix autogen/go mod

* Handling resource extraction when custom metrics are allowed

* Add tests for AllowCustomMetrics set to true

* Address vagababov and yanweiguo comments.

Signed-off-by: Evan Anderson <evan.k.anderson@gmail.com>

* address comments

* Renaming

* Reduce some condition nesting

* go fmt

* Sleep for 1ms to allow metrics being sent (#9)

* Address yanweiguo comments

Co-authored-by: JJ Zeng <jjzeng@google.com>
Co-authored-by: jjzeng-seattle <56568376+jjzeng-seattle@users.noreply.github.com>
2020-07-14 10:46:19 -07:00
Victor Agababov 34c17cc2d5
Make things private that should be so (#1417)
* Make things private that should be so

Make some constants and enums private.
The ones I found to be used elsewhere, I kept public.

* sort
2020-06-17 09:26:24 -07:00
Antoine Cotten ffb929374a
Make Prometheus exporter port configurable via env (#1411) 2020-06-16 16:26:24 -07:00
Antoine Cotten 0d967ff0d8
Clean metrics config tests (#1412)
* Use constants in metrics config tests

* Normalize test cases indentation
2020-06-16 11:09:24 -07:00
Yanwei Guo f1c51d6830
Delete nit check log (#1340)
* do not record for empty metric config

* Revert "do not record for empty metric config"

This reverts commit 539a5e4dbb.

* delete
2020-05-16 09:00:02 -07:00
Yanwei Guo 16d7b96341
Print nil check log only once (#1334)
* do not record for empty metric config

* Revert "do not record for empty metric config"

This reverts commit 539a5e4dbb.

* delete

* log once

* remove space
2020-05-14 17:25:00 -07:00
Yanwei Guo de5c590700
Don't record data points when metric config is not initialized yet (#1323)
* do not record for empty metric config

* Revert "do not record for empty metric config"

This reverts commit 539a5e4dbb.

* dont record

* add comment
2020-05-11 15:34:46 -07:00
Annie Fu 099343d49e
Make updating metrics config and metrics exporter atomic (#1271)
* Make updating metricsConfig and updating metricsExporter atomic

This requires doing anything with a kubeclient before obtaining the
metricsMux lock.

* Address PR
2020-05-01 15:35:42 -07:00
Markus Thömmes 9b70d057b1
Allow a 'None' metrics backend, to basically do nothing. (#1190) 2020-04-04 11:17:27 -07:00
Evan Anderson 7af3fab62c
Roll forward #1045 (#1109)
* Revert "Add support for client TLS to pkg/metrics (#1045)"

This reverts commit 945b556708.

* Roll forward "Add support for client TLS to pkg/metrics (#1045)"

Adds support for client TLS certs for opencensus export

* Switch sharedmain to only pass a getter rather than an all-namespace lister.

* Add a TODO about using a cached copy if this generates undue load

* Update deps per build failure.

* Refactor per @anniefu suggestion
2020-02-25 11:42:51 -08:00
Evan Anderson 1ad2b11321
Revert "Add support for client TLS to pkg/metrics (#1045)" (#1106)
This reverts commit 945b556708.
2020-02-19 08:20:06 -08:00
Evan Anderson 945b556708
Add support for client TLS to pkg/metrics (#1045)
* Document the plan for metrics

* Update with issue links

* Update with other @mattmoor comments.

* Address feedback from @anniefu

* Update export address since `localhost` won't work for a DaemonSet.

* Add an explicit action item to migrate Prometheus

* Fix possible (ambiguous) link vs checkbox.

* Fix typo in metrics/README.md

* Add support for client TLS to pkg/metrics

* Address comments from @vagababov and @anniefu

* Finish removing Resource code

* Update name to be shorter and not require a double migration.

* Add tests for opencensus exporting with TLS
2020-02-18 11:04:58 -08:00
Victor Agababov 92c9752a4c
Add RecordN method to permit recording of N measurements simultaneously. (#1025)
* Add RecordN method to permit recording of N measurements simultaneously.

In serving we have many places where we try to record several metrics at the same time.
Besides very repetetive code, this also brings inefficiences, since metrics code has to do _some_ work
for every call, which is completely avoidable.

This way we can record N measurements with one hit.

* tests and nits

* review items per discussion with Evan:

- remove ros
- make measurements ...

* review
2020-01-31 11:26:28 -08:00
Evan Anderson 8c11c24324 Add support for OpenCensus Agent as a destination for Knative metrics. (#978)
* Add support for OpenCensus Agent as a destination for Knative metrics.

* Update with comments from @mattmoor and @anniefu

* Remember to register exporter.

Fix tests.

* Fix naming per discussion with @anniefu
2020-01-10 18:42:58 -08:00
Ignacio (Nacho) Cano daef119f73 adding knative_broker metrics (#944) 2020-01-06 15:21:49 -08:00
Matt Moore e96196e347 golang format tools (#876)
Produced via:
  `gofmt -s -w $(find -path './vendor' -prune -o -type f -name '*.go' -print))`
  `goimports -w $(find -name '*.go' | grep -v vendor)`
/assign n3wscott
2019-11-13 15:49:28 -08:00
Nima Kaviani 4deb5d83d2 move metrics config to pkg (#786)
the observability config is used in other packages including
the sample controller. This commit pulls observability config
into knative.dev/pkg similar to logging config for better reusability.
2019-11-13 06:52:28 -08:00
Evan Anderson a9bdd123b7 Start moving Stackdriver and Prometheus-specific code to a strategy pattern (#812)
* Start moving Stackdriver and Prometheus-specific code to a strategy pattern.

* Respond to review feedback, add test.

* gofmt
2019-11-04 16:05:21 -08:00
Annie Fu 4d39b058e2 Restrict metric pkg Stackdriver Secret to controlled name & namespace (#824)
* Restrict Stackdriver Secret to controlled name & namespace

* Address PR comments

* Lock when reading secret name & namespace

* Export StackdriverClientConfig struct so consumers don't have to
redefine it

* Ensure metrics pkg consumers call SetStackdriverSecretLocation to set
Secret name and namespace

* Export StackdriverClientConfig consts & helper function

* Add TODO for secret watcher
2019-11-04 11:57:19 -08:00
Annie Fu f684a1078a Add field to metrics config for configuring Stackdriver exporter client (#794)
* Add field to metrics config for configuring Stackdriver exporter client
options

* Fix test setting global state and interfering with other tests

* Use correct GCP metadata values & cleanup

* Some code cleanup

* Remove as much test-specific function stubbing as possible

* Remove more stubbed out test code

* Make more descriptive error when getting secret

* Use former locking pattern to avoid deadlock

There is a cyclical dependency between the metrics pkg and client-go
kubeclient where the kubeclient will attempt to read the metricsConfig
to record metrics about kubeclient requests. This prevents using a kubeclient within the metricsMux lock.
2019-10-25 10:28:36 -07:00
Annie Fu 66c91bcfa6 Fix race condition when updating Stackdriver exporter (#769)
* Remove race condition between reading updates to config-observability
and updating stackdriver exporter

* Simply isNewExporterRequired if statement

* Remove unused thread-safe versions of functions

* Use a reader-writer lock & add function comment

* Make internal unlocked function names consistent
2019-10-18 13:48:19 -07:00
Yanwei Guo 9a71140445 add comment (#761) 2019-10-14 15:29:32 -07:00
Ignacio Cano 6b1c5cf09a StackDriver CustomMetrics with Custom Domain Prefix (#676)
* renaming source_name and source_resource_group to name and resource_group,
as they will be used in other resources

* updating

* updates based on code review

* update

* fix
2019-09-13 15:50:35 -07:00
Ignacio Cano c56cedc7e7 Serializing/Deserializing Logging and Metrics configs (#656)
* source stats reporter... should be used by well-behaved sources

* public fields

* moving serialization/deserialization of metrics and logging maps to pkg

* nits

* adding UT

* same order

* updates

* sock-puppet

* unregistration problem?

* removing base64 encoding

* removing TODO

* comments after review

* removing the other constants
2019-09-10 12:36:29 -07:00
Matt Moore 222dd25986 Migrate pkg to use the knative.dev/pkg import path (#489)
* Manual changes.

* scripted changes.
2019-06-26 13:02:06 -07:00
Matt Moore 523d91657b Add a testing package to setup metrics.Domain() (#429)
This mirrors what we do for system.Namespace() today.
2019-05-28 08:01:41 -07:00
Matt Moore 2923ca42d9 Share the configmap name logic for Logging and Metrics. (#425)
This also adds a way to fetch the metrics "domain" following a similar pattern to `system.Namespace()`.
2019-05-27 18:41:29 -07:00
Mustafa Demirhan 3c8c4a9354 Add support to flush the metrics exporter (#391)
* Add support to flush the metrics exporter.

* Updates to the comments.
2019-04-18 20:12:27 -07:00
Yanwei Guo 5323eaf18b Expose UpdateExporter function to accept map as parameter (#347)
* expose func

* use different metric name

* remove stackdriver change

* change comment

* return err
2019-03-29 16:38:54 -07:00
Yanwei Guo b4a3bde045 Decide whether to record metrics according to current metrics config (#292)
* split and update

* bug fixed

* change the copyright year

* use fake metadata in tests

* exclude test coverage for gcp_metadata

* add record

* add tests

* revert 2019 and add a new var

* change documents

* address comment

* change line
2019-03-07 10:26:42 -08:00
Victor Agababov c100323403 Fix the sugared logger usage (#300)
* Fix the sugared logger usage

Without `w` the logged messages are quite useless and annoying.

* some more magic
2019-02-28 18:17:38 -08:00
Yanwei Guo 1aca435944 Read metrics backend from environment variable (#218)
* add env to set backend

* join two lines
2019-01-10 11:03:42 -08:00
jonjohnsonjr a8c40a9e3b Default metrics.backend-destination to Prometheus (#213)
Fixes #212
2019-01-08 09:54:43 -08:00
Mustafa Demirhan 53b1235c2a Use different reporting periods based on the metrics backend (#182)
* Use different reporting periods based on the metrics backend. Original implementation was setting this value to 1 min to accommodate the minimum allowed value for stackdriver, but that is a sub optimal behavior for Prometheus. The change also allows operator to override the default values.

* Fix the units of reconciler latency from nano seconds to milliseconds.

* Address PR comments.

* Add better logging for reconciliation.

* Remove unnecessary lock in the UT.

* Address PR comments.
2018-12-04 15:16:25 -08:00
akyyy 02b354e3a4 Relax the requirement for stackdriver project id for metrics exporter. (#147)
* fix flaky test

* relax the requirement for stackdriver project id

* improve log

* add comment for stackdriverProjectIDKey not provided case

* minor format

* simplify the map access
2018-10-30 16:20:34 -07:00