abc3097d75
* upgrade to latest dependencies bumping k8s.io/kube-openapi 4241196...3ee0da9: > 3ee0da9 Merge pull request # 299 from Jefftree/def-optimize > 3a31a64 Merge pull request # 292 from Jefftree/v2v3conv > ba6f675 Optimize Definition generation > e69a86c Merge pull request # 298 from Jefftree/allOf > 2cbad14 Add OpenAPIConv > 9f9c01d Merge pull request # 297 from cici37/errFix > 207f0c7 Wrap Refs with AllOf > 29d7264 Merge pull request # 295 from Jefftree/fix-api-json > d2a55e8 Align maxProperties/minProperties err value with others > 52feaf8 Merge pull request # 290 from jiahuif/feature/prune-v2-when-serving-v3 > 352e363 Fix json encoding for OpenAPI v3 Discovery > 662cbb0 Return valid for kubernetes to handle err properly > 13192b9 Merge pull request # 289 from jiahuif/feature/openapi-oneof-type > 2580423 remove embedded v2 from expected schema > 8a7ee80 Merge pull request # 293 from alexzielenski/cache-busting > 79a5a7d generated: update integration test. > d436835 add test for pruning v2 schema. > 6a7b704 Merge pull request # 291 from Jefftree/builder3-test > de6f8b0 split openapi mock class into Fake and FakeV3 > da7e27a add integration test for v3 OneOf > eca4f62 do not include v2 schema when serving v3. > 89ac9db Merge pull request # 284 from Jefftree/cache-busting > 7afa56c Add test for builder3 > 3ca26f0 address comments > 697543f add support for custom v3 oneOf types. > ddc6692 Merge pull request # 288 from Jefftree/gnostic-fix > b0d7548 Cache busting > 130db2b add wrapper around gnostic v3 document > a9dfcaa Merge pull request # 286 from Jefftree/leading-newline > 91ab739 Update gnostic to drop jsonschema dependency > 3f90b8c Merge pull request # 282 from alexzielenski/header_vendorext_marshal > c7e0de3 Merge pull request # 285 from Jefftree/proto-upgrade > 7626df5 Trim leading newlines in Descriptions > 1cd4920 add vendor extensions to marshal openapi v2 header > 86aaf54 Upgrade protobuf and change references of googleapis/gnostic -> google/gnostic Signed-off-by: Knative Automation <automation@knative.team> * Fix client-go version * Add missing license file * Add another missing license file Signed-off-by: Knative Automation <automation@knative.team> Co-authored-by: David Simansky <dsimansk@redhat.com> |
||
|---|---|---|
| .. | ||
| metricskey | ||
| README.md | ||
| client.go | ||
| config.go | ||
| config_observability.go | ||
| doc.go | ||
| exporter.go | ||
| memstats.go | ||
| metrics.go | ||
| metrics_worker.go | ||
| opencensus_exporter.go | ||
| prometheus_exporter.go | ||
| record.go | ||
| resource_view.go | ||
| testing.go | ||
| utils.go | ||
| workqueue.go | ||
| zz_generated.deepcopy.go | ||
README.md
Common metrics export interfaces for Knative
See the Plan for details on where this is heading.
Current status
The code currently uses OpenCensus to support exporting metrics to multiple backends. Currently, two backends are supported: Prometheus and OpenCensus/OTel.
Metrics export is controlled by a ConfigMap called config-observability which
is a key-value map with specific values supported for each of the OpenCensus
and Prometheus backends. Hot-reload of the ConfigMap on a running process is
supported by directly watching (via the Kubernetes API) the
config-observability object. Configuration via environment is also supported
for use by the queue-proxy, which runs with user permissions in the user's
namespace.
Problems
There are currently
6 supported Golang exporters for OpenCensus.
We do not want to build all of those backends into the core of knative.dev/pkg and all
downstream dependents, and we'd like all the code shipped in knative.dev/pkg
to be able to be tested without needing any special environment setup.
With the current direct-integration setup, there needs to be initial and ongoing
work in pkg (which should be high-value, low-churn code) to maintain and
update stats exporters which need to be statically linked into ~all Knative
binaries. This setup also causes problems for vendors who may want or need to
perform an out-of-tree integration (e.g. proprietary or partially-proprietary
monitoring stacks).
Another problem is that each vendor's exporter requires different parameters,
supplied as Golang Options methods which may require complex connections with
the Knative ConfigMap. Two examples of this are secrets like API keys and the
Prometheus monitoring port (which requires additional service/etc wiring).
See also this doc, where the plan was worked out.
The plan
OpenCensus (and eventually OpenTelemetry) offers an sidecar or host-level agent with speaks the OpenCensus protocol and can proxy from this protocol to multiple backends.
(From OpenCensus Documentation)
We will standardize on export to the OpenCensus export protocol, and encourage
vendors to implement their own OpenCensus Agent or Collector DaemonSet, Sidecar,
or other
OpenCensus Protocol
service which connects to their desired monitoring environment. For now, we
will use the config-observability ConfigMap to provide the OpenCensus
endpoint, but we will work with the OpenTelemetry group to define a
kubernetes-friendly standard export path.
Additionally, once OpenTelemetry agent is stable, we will propose adding the OpenTelemetry agent running on a localhost port as part of the runtime contract.
We need to make sure that the OpenCensus library does not block, fail, or queue metrics in-process excessively in the case where the OpenCensus Agent is not present on the cluster. This will allow us to ship Knative components which attempt to reach out the Agent if present, and which simply retain local statistics for a short period of time if not.
Concerns
- Unsure about the stability of the OpenCensus Agent (or successor). We're currently investigating this, but the OpenCensus agent seems to have been recommended by several others.
- Running
fluentdas a sidecar was very big (400MB) and had a large impact on cold start times.- Mitigation: run the OpenCensus agent as a DaemonSet (like we do with
fluentdnow).
- Mitigation: run the OpenCensus agent as a DaemonSet (like we do with
- Running as a DaemonSet may make it more difficult to ensure that metrics for
each namespace end up in the right place.
- We have this problem with the built-in configurations today, so this doesn't make the problem substantially worse.
- May want/need some connection between the Agent and the Kubelet to verify sender identities eventually.
- Only expose OpenCensus Agent on localhost, not outside the node.
Steps to reach the goal
- Add OpenCensus Agent as one of the export options.
- Ensure that all tests pass in a non-Google-Cloud connected environment. This is true today. Ensure this on an ongoing basis.
- Google to implement OpenCensus Agent configuration to match what they are doing for Stackdriver now. (No public issue link because this should be in Google's vendor-specific configuration.)
- Document how to configure OpenCensus/OpenTelemetry Agent + Prometheus to achieve the current level of application visibility, and determine a long-term course for how to maintain this as a "bare minimum" supported configuration. https://github.com/knative/docs/pull/3005
- Stop adding exporter features outside of the OpenCensus / OpenTelemetry export as of 0.13 release (03 March 2020). Between now and 0.13, small amounts of additional features can be built in to assist with the bridging process or to support existing products. New products should build on the OpenCensus Agent approach.
- Removal of the Stackdriver OpenCensus Exporter https://github.com/knative/pkg/issues/2173
- Revisit Adopting OpenTelemetry SDKs instead of OpenCensus