Commit Graph

68 Commits

Author SHA1 Message Date
Douglas Reid 4f4173257d
stackdriver(logging): add support for cel logging filters (#3630)
* stackdriver(logging): add support for cel logging filters

* check if filter expression exists in init
2022-01-06 12:31:52 -08:00
Douglas Reid e777f05cdc
feat(stackdriver): add more detailed logging around overrides (#3580) 2021-11-10 15:28:24 -08:00
Douglas Reid ff775f5605
Add support for configurability of stackdriver metrics (#3505)
* add support for configurability of stackdriver metrics

* add tag validity checks and support apiName and apiVersion tags

* cleanup

* address review comments

* more const refs

* update config doc

* update const refs

* push tagkey registration into configure()
2021-09-30 22:17:14 -07:00
Pengyuan Bian 3a9cc09047
Update envoy SHA to 6/18 (#3387)
* wip std -> absl

* update.

* wip

* update.

* update

* fix.

* update

* update

* clean

* fix
2021-06-29 16:11:26 -07:00
Pengyuan Bian f1bacaa7dd
Remove Edge reporting. (#3379)
* remove edge

* build.

* fix

* update

* update.

* update.
2021-06-18 10:59:36 -07:00
Kuat baf58ffb3b
update envoy (#3376)
* update envoy

Signed-off-by: Kuat Yessenov <kuat@google.com>

* fix format

Signed-off-by: Kuat Yessenov <kuat@google.com>
2021-06-14 22:51:39 -07:00
Yangmin Zhu 0efe658755
fix the stackdriver logging for dry-run policy (#3361)
* fix the stackdriver logging for dry-run policy

* address comments
2021-05-20 21:07:01 -07:00
Yangmin Zhu 806fdda01e
authz: add initial authz dry-run implementation (#3256)
* authz: add initial authz dry-run implementation

* fix tsan

* add tcp test

* do not use regex
2021-03-24 11:57:10 -07:00
Pengyuan Bian 4d6782f636
Add configuration for stackdriver metric expiry. (#3134)
* Set sd metric reporting expiry to 1 hour.

* add configurability

* fix

* update

* add an integration test

* update
2021-01-13 12:19:12 -08:00
Nupur Garg 45ef14104a
Add customized access logging support for stackdriver access logs (#3047)
* Add customized access logging support for stackdriver access logs

TODO: add support for removal of tags and customization of TCP access logs

Improve comment

Remove proto refactoring
Enable it for TCP

fix extra semi colon

add cleanupExpressions and cache for TCP

fix test

fix comments

* fix build

* fix comments

* Update doc
2020-10-20 17:59:42 -07:00
Kuat eec9ae9f4d
fix: implement metadata fallback on the special value (#3056)
* fix handling of peer metadata for TCP

Signed-off-by: Kuat Yessenov <kuat@google.com>

* edit

Signed-off-by: Kuat Yessenov <kuat@google.com>
2020-10-19 13:28:51 -07:00
Pengyuan Bian 58f07a7722
Parse workload and service from resource labels (#3051)
* update sha

* parse workload and service from resource labels

* add test

* clean up

* fix

* clean up

* skip looking at localhost endpoint

* fix

* fix

* update cluster name
2020-10-15 17:55:12 -07:00
Nupur Garg a5ef9f7daf
Minor changes to Access Logs (#3000)
1) Made batch size and interval for reporting configurable
2) Don't add labels if they are empty-> this is done for labels that are known to empty sometimes
3) Add upstream_transport_failure_reason label to logs

Signed-off-by: gargnupur <gargnupur@google.com>

Added a default timer for ticker

Signed-off-by: gargnupur <gargnupur@google.com>

Fixed feedback

Signed-off-by: gargnupur <gargnupur@google.com>

Updated

Signed-off-by: gargnupur <gargnupur@google.com>
2020-10-08 08:59:47 -07:00
Kuat 938a9485a4
update envoy-wasm (#3032)
* export

Signed-off-by: Kuat Yessenov <kuat@google.com>

* fix build

Signed-off-by: Kuat Yessenov <kuat@google.com>

* add an empty test

Signed-off-by: Kuat Yessenov <kuat@google.com>

* fix build

Signed-off-by: Kuat Yessenov <kuat@google.com>

* egregious logging

Signed-off-by: Kuat Yessenov <kuat@google.com>

* use detached buffer

Signed-off-by: Kuat Yessenov <kuat@google.com>

* cleanup

Signed-off-by: Kuat Yessenov <kuat@google.com>

* review

Signed-off-by: Kuat Yessenov <kuat@google.com>
2020-10-06 14:25:16 -07:00
Pengyuan Bian a7ef110ba4
Update stackdriver filter to fallback to default mesh id format when not specified (#2993)
* update stackdriver filter to fallback to default mesh id format when not specified

* skip setting mesh id if project number if not known

* fix comment

* fix

* fix

* fix test

* fix test again
2020-08-28 19:11:31 -07:00
David Raskin 40c3904827
Add audit log stream to stackdriver filter (#2970)
* Initial auditEntry implementation

Signed-off-by: davidraskin <draskin@google.com>

* Initial audit log implementation

Signed-off-by: davidraskin <draskin@google.com>

* Format

Signed-off-by: davidraskin <draskin@google.com>

* Use LogEntryType and map for WriteLogEntriesRequest

Signed-off-by: davidraskin <draskin@google.com>

* Add function to initialize logEntriesRequests

Signed-off-by: davidraskin <draskin@google.com>

* Format

Signed-off-by: davidraskin <draskin@google.com>

* Switched to using bool for outbound and bool for audit

Signed-off-by: davidraskin <draskin@google.com>

* Format and remove uneeded functions

Signed-off-by: davidraskin <draskin@google.com>

* Undo stats modification for non cpp20

Signed-off-by: davidraskin <draskin@google.com>

* Takeout double setting of label

Signed-off-by: davidraskin <draskin@google.com>

* Update extensions/common/context.h

Co-authored-by: Nupur Garg <37600866+gargnupur@users.noreply.github.com>

* Update extensions/common/context.h

Co-authored-by: Nupur Garg <37600866+gargnupur@users.noreply.github.com>

* Remove redundant request_info population

Signed-off-by: davidraskin <draskin@google.com>

* Addressed comments

Signed-off-by: davidraskin <draskin@google.com>

* Update comments + change when audit entry is added

Signed-off-by: davidraskin <draskin@google.com>

* Update comments. remove unnecessary include

Signed-off-by: davidraskin <draskin@google.com>

* format

Signed-off-by: davidraskin <draskin@google.com>

* Fix tcp audit logging

Signed-off-by: davidraskin <draskin@google.com>

* Documentation of config

Signed-off-by: davidraskin <draskin@google.com>

* Integration test + address comments

Signed-off-by: davidraskin <draskin@google.com>

* Format

Signed-off-by: davidraskin <draskin@google.com>

* test multiple auditentries + format comment

Signed-off-by: davidraskin <draskin@google.com>

* Remove test case list + add test to inventory

Signed-off-by: davidraskin <draskin@google.com>

Co-authored-by: Nupur Garg <37600866+gargnupur@users.noreply.github.com>
2020-08-20 10:37:12 -07:00
Nupur Garg 3b0c7f243f
Enable Client Side Access Logs for SD (#2955)
* Enable Client Side Access Logs for SD

Fix fmt

Fix fmt

Fix test

Added config options and test for the same

Fixed after rebase

Fixed config and added another test case

Run fmt

Change from ERR_ONLY to ERR_ONLY_ON_NO_MX

Fixed based on feedback

Updated config

Updated config

Fixed based on feedback

* Fixed based on feedback

* Fix lint

* change client error access log
2020-08-11 12:05:47 -07:00
Nupur Garg 35481921f7
Add option to enable Log Compression for stackdriver access logs (#2919)
* parent c51fe751a1
author gargnupur <gargnupur@google.com> 1594947318 +0000
committer gargnupur <gargnupur@google.com> 1595279382 -0700

Add option to enable Log Compression for stackdriver access logs

Update comment

Format

Update comment

Change it to a proto config

Use BoolValue

Use BoolValue

Remove file added by mistake

* fix fmt

Signed-off-by: gargnupur <gargnupur@google.com>
2020-07-21 00:49:06 -07:00
Nupur Garg 47d38c13e1
Add TCP Access Log on Open (#2912)
* Add TCP Access Log on Open

Signed-off-by: gargnupur <gargnupur@google.com>

fixed formatting

Signed-off-by: gargnupur <gargnupur@google.com>

Adding to see exact thing in PR for my own review

Signed-off-by: gargnupur <gargnupur@google.com>

Adding to see exact thing in PR for my own review -> fix formatting

Signed-off-by: gargnupur <gargnupur@google.com>

Fix tests

Signed-off-by: gargnupur <gargnupur@google.com>

Fix format

Signed-off-by: gargnupur <gargnupur@google.com>

Fix test

Signed-off-by: gargnupur <gargnupur@google.com>

* make tests parallel

Signed-off-by: gargnupur <gargnupur@google.com>

* fix lint error

Signed-off-by: gargnupur <gargnupur@google.com>

* fix test and based on feedback

Signed-off-by: gargnupur <gargnupur@google.com>

* Add log_sampled after rebase

Signed-off-by: gargnupur <gargnupur@google.com>

* Log metrics too on timeout

Signed-off-by: gargnupur <gargnupur@google.com>

* Fix based on feedback

Signed-off-by: gargnupur <gargnupur@google.com>
2020-07-20 21:33:29 -07:00
Nupur Garg 5f16bfd2dc
Add filtered label for sampled logs (#2918)
* Add filtered label for sampled logs

Signed-off-by: gargnupur <gargnupur@google.com>

* Fix unit test

Signed-off-by: gargnupur <gargnupur@google.com>

* Fix unit test

Signed-off-by: gargnupur <gargnupur@google.com>

* Fixed based on feedback

Signed-off-by: gargnupur <gargnupur@google.com>

* Fix based on feedback

Signed-off-by: gargnupur <gargnupur@google.com>
2020-07-17 18:54:34 -07:00
Nupur Garg ea0ce55278
Add TCP Access Logging (#2852)
* Add TCP Metrics in Stackdriver

Signed-off-by: gargnupur <gargnupur@google.com>

fix

Signed-off-by: gargnupur <gargnupur@google.com>

changes

Signed-off-by: gargnupur <gargnupur@google.com>

refactor

Signed-off-by: gargnupur <gargnupur@google.com>

fix test

Add TCP Metrics in Stackdriver

Signed-off-by: gargnupur <gargnupur@google.com>

fix

Signed-off-by: gargnupur <gargnupur@google.com>

Add TCP access logging

changes

Signed-off-by: gargnupur <gargnupur@google.com>

Remove local path from Makefile

changes

Signed-off-by: gargnupur <gargnupur@google.com>

Add functions for queue

fix build

fix test

remove debug time

Signed-off-by: gargnupur <gargnupur@google.com>

remove debug time

Signed-off-by: gargnupur <gargnupur@google.com>

Add protocol in http accesslog

* fix lint

Signed-off-by: gargnupur <gargnupur@google.com>

* fix merge from master

Signed-off-by: gargnupur <gargnupur@google.com>

* fixed based on feedback

Signed-off-by: gargnupur <gargnupur@google.com>

* fix test

* fixed based on feedback

Signed-off-by: gargnupur <gargnupur@google.com>

* add comment
2020-06-08 11:56:57 -07:00
Nupur Garg 46babbff42
Add TCP Metrics in Stackdriver (#2843)
* Add TCP Metrics in Stackdriver

Signed-off-by: gargnupur <gargnupur@google.com>

fix

Signed-off-by: gargnupur <gargnupur@google.com>

changes

Signed-off-by: gargnupur <gargnupur@google.com>

refactor

Signed-off-by: gargnupur <gargnupur@google.com>

fix test

Fixed based on feedback

Signed-off-by: gargnupur <gargnupur@google.com>

* Fix yaml files

Signed-off-by: gargnupur <gargnupur@google.com>

* Fixed based on feedback

Signed-off-by: gargnupur <gargnupur@google.com>

* Fixed based on feedback

Signed-off-by: gargnupur <gargnupur@google.com>

* Set timer always

Signed-off-by: gargnupur <gargnupur@google.com>

* remove access logger for TCP

Signed-off-by: gargnupur <gargnupur@google.com>
2020-06-04 10:15:56 -07:00
Pengyuan Bian 93937b6861
Switch back envoy upstream to envoy-wasm (#2862)
* switch back to envoy wasm

* remove envoy path override in check_wasm job

* fix release build

* fix release build

* fix wasm build target

* disable rbe

* fix build and restore rbe

* clean up

* format

* add bazel config rel

* fix wasm path again

* add resource class to circle macos flow

* remove large resource class and test step
2020-05-27 07:55:55 -07:00
mandarjog 2379bf3da9
ensure that telemetry plugins do not return config errors (#2857)
* ensure that telemetry plugin do not return errors

* initialize to false
2020-05-19 10:48:07 -07:00
Kuat d665d3d02d
Remove protobuf from stats extension (#2807)
* fix dependencies

Signed-off-by: Kuat Yessenov <kuat@google.com>

* finish

Signed-off-by: Kuat Yessenov <kuat@google.com>

* typos

Signed-off-by: Kuat Yessenov <kuat@google.com>

* reformat

Signed-off-by: Kuat Yessenov <kuat@google.com>

* reset tests

Signed-off-by: Kuat Yessenov <kuat@google.com>

* merge fixes

Signed-off-by: Kuat Yessenov <kuat@google.com>

* wip

Signed-off-by: Kuat Yessenov <kuat@google.com>

* wip

Signed-off-by: Kuat Yessenov <kuat@google.com>

* wip

Signed-off-by: Kuat Yessenov <kuat@google.com>

* fix

Signed-off-by: Kuat Yessenov <kuat@google.com>

* fixes

Signed-off-by: Kuat Yessenov <kuat@google.com>

* fixes

Signed-off-by: Kuat Yessenov <kuat@google.com>

* fixes

Signed-off-by: Kuat Yessenov <kuat@google.com>

* fix wasm

Signed-off-by: Kuat Yessenov <kuat@google.com>

* print file sizes

Signed-off-by: Kuat Yessenov <kuat@google.com>

* review

Signed-off-by: Kuat Yessenov <kuat@google.com>

* fix up parsing

Signed-off-by: Kuat Yessenov <kuat@google.com>

* tolerate failure

Signed-off-by: Kuat Yessenov <kuat@google.com>
2020-04-24 16:54:29 -07:00
Kuat 2b0dd1cd5c
update envoy to 1.14.1 (#2817)
* fixes to update envoy

Signed-off-by: Kuat Yessenov <kuat@google.com>

* build fix

Signed-off-by: Kuat Yessenov <kuat@google.com>

* fix wasm build

Signed-off-by: Kuat Yessenov <kuat@google.com>

* fix local string leak

Signed-off-by: Kuat Yessenov <kuat@google.com>
2020-04-22 16:24:34 -07:00
Douglas Reid 02dcb15307
fix(stackdriver): support turning off http size metrics (#2767)
* fix(stackdriver): support turning off client metrics and http size metrics

Signed-off-by: Douglas Reid <douglas-reid@users.noreply.github.com>

* remove client-side metrics option and add tests

Signed-off-by: Douglas Reid <douglas-reid@users.noreply.github.com>

* rebase

* clang-format

* refactor to fix asan issues
2020-04-20 12:46:36 -07:00
Pengyuan Bian 7771aefa2a
Consolidate proxy tests to use step based framework (#2806)
* consolidate proxy tests to use step based framework

* lint

* extend latency boundary for asan tsan build

* sleep one sec before sending request in access log test

* prolong export period for stackdriver test to bear asan/tsan initial loading time

* address comment

* sleep a bit after envoy starting up
2020-04-16 18:04:24 -07:00
Kuat fcec9c0bf8
cleanup: remove unused code (#2802)
* remove stale code

Signed-off-by: Kuat Yessenov <kuat@google.com>

* simplify

Signed-off-by: Kuat Yessenov <kuat@google.com>

* cleanup

Signed-off-by: Kuat Yessenov <kuat@google.com>

* fix linter

Signed-off-by: Kuat Yessenov <kuat@google.com>

* fix linter

Signed-off-by: Kuat Yessenov <kuat@google.com>
2020-04-14 19:39:30 -07:00
Kuat 910639ad62
replace protobuffers with flatbuffers at runtime (#2794)
* implement flatbuffer read and measure

Signed-off-by: Kuat Yessenov <kuat@google.com>

* fix all tests

Signed-off-by: Kuat Yessenov <kuat@google.com>

* add cache

Signed-off-by: Kuat Yessenov <kuat@google.com>

* finish

Signed-off-by: Kuat Yessenov <kuat@google.com>

* fix build?

Signed-off-by: Kuat Yessenov <kuat@google.com>

* maybe?

Signed-off-by: Kuat Yessenov <kuat@google.com>

* maybe?

Signed-off-by: Kuat Yessenov <kuat@google.com>

* simplify

Signed-off-by: Kuat Yessenov <kuat@google.com>

* feedback

Signed-off-by: Kuat Yessenov <kuat@google.com>

* fix wasm build

Signed-off-by: Kuat Yessenov <kuat@google.com>

* set peer size by default

Signed-off-by: Kuat Yessenov <kuat@google.com>

* split inbound and outbound peer caches

Signed-off-by: Kuat Yessenov <kuat@google.com>

* merge fix

Signed-off-by: Kuat Yessenov <kuat@google.com>
2020-04-08 11:22:55 -07:00
Pengyuan Bian 6fee8ca5dc
Improve Wasm plugins stats (#2783)
* fix plugin metrics

* format

* move stats to a common place

* add test

* fix format

* fix test

* fix

* rename stats

* rewrite comments
2020-03-31 14:01:21 -07:00
Pengyuan Bian c785a47b22
Add configurable monitoring endpoint (#2785)
* add configurable monitoring endpoint

* add monitoring endpoint override to sts
2020-03-26 10:58:56 -07:00
Pengyuan Bian d7d2ad8100
Flush log entries in on done (#2780)
* flush log entries in on done

* add a comment about logger

* clean up logging stats since it is already tracked by envoy grpc stats

* simplify vm reload test
2020-03-24 14:31:55 -07:00
Pengyuan Bian 7598da631d
add x-goog-user-project header to stackdriver call out (#2731)
* add test for x-goog-user-project header

* add x-goog-user-project header to stackdriver callout
2020-03-23 15:19:11 -07:00
Pengyuan Bian 0237b2392e
Add stackdriver stub param (#2739)
* add stackdriver stub param

* dedup envoy grpc proto build code

* clean up
2020-03-03 15:22:59 -08:00
Kuat 71b7a6cc2c
allow unknown fields in configs (#2734)
Signed-off-by: Kuat Yessenov <kuat@google.com>
2020-02-28 10:46:44 -08:00
Douglas Reid 9faeb5fc89
feat(edges): support configurable batch size (#2707)
* feat(edges): support configurable batch size

Signed-off-by: Douglas Reid <douglas-reid@users.noreply.github.com>

* address review comments

Signed-off-by: Douglas Reid <douglas-reid@users.noreply.github.com>

* address review comments

Signed-off-by: Douglas Reid <douglas-reid@users.noreply.github.com>
2020-02-25 15:03:21 -08:00
Kuat 3a04f855cd
test: STS integration with Stackdriver (#2700)
* implement STS test

Signed-off-by: Kuat Yessenov <kuat@google.com>

* fix lint

Signed-off-by: Kuat Yessenov <kuat@google.com>

* regenerate wasm

Signed-off-by: Kuat Yessenov <kuat@google.com>
2020-02-20 23:51:56 -08:00
Douglas Reid 748ac4cf51
feat(edges): add support for reporting two batches (new vs full) (#2669)
* feat(edges): add support for reporting two batches (new vs full)

Signed-off-by: Douglas Reid <douglas-reid@users.noreply.github.com>

* cleanup

Signed-off-by: Douglas Reid <douglas-reid@users.noreply.github.com>

* fix comment in test

Signed-off-by: Douglas Reid <douglas-reid@users.noreply.github.com>

* address review feedback

Signed-off-by: Douglas Reid <douglas-reid@users.noreply.github.com>
2020-02-13 10:24:44 -08:00
Douglas Reid cdf36928ee
feat(stats): support grpc status codes in metrics (#2624)
* feat(stats): support grpc status codes in metrics

* wip

* add tests and fix up context

* set empty grpc_response_code

* use latest envoyproxy/envoy-wasm

Signed-off-by: Douglas Reid <douglas-reid@users.noreply.github.com>

* add license/copyright banner

Signed-off-by: Douglas Reid <douglas-reid@users.noreply.github.com>

* fix lint / format / malign issues

Signed-off-by: Douglas Reid <douglas-reid@users.noreply.github.com>

* fix up alpn_test.cc

Signed-off-by: Douglas Reid <douglas-reid@users.noreply.github.com>

* fix lint

Signed-off-by: Douglas Reid <douglas-reid@users.noreply.github.com>

* more tests needed updating with envoy update

Signed-off-by: Douglas Reid <douglas-reid@users.noreply.github.com>

* stackdriver fix

Signed-off-by: Douglas Reid <douglas-reid@users.noreply.github.com>

* fix stackdriver onConfigure

Signed-off-by: Douglas Reid <douglas-reid@users.noreply.github.com>

* remove unused using clause

Signed-off-by: Douglas Reid <douglas-reid@users.noreply.github.com>

* clang-format

Signed-off-by: Douglas Reid <douglas-reid@users.noreply.github.com>
2020-02-05 16:54:33 -08:00
Pengyuan Bian f047451c54
Add sts to sd filter (#2638)
* add sts to sd filter

* fix sts token path

* fix format

* update metric channle credential

* clean up

* extract sts grpc filling logic

* make sts options constant

* using localhost instead

* fix

* format
2020-01-31 19:38:35 -08:00
Nupur Garg b36332b065
Add TCP Metrics in stats filter similar to Istio 1.4 (#2629)
* Add TCP Metrics in Istio 1.5 similar to Istio 1.4

Fix wasm_proxy test

Add timer based TCP metrics

Fix lint err

Fix lint err

Regenerate WASM file

Remove locks as WASM is single threaded

Add plugin.wasm

Fixed based on feedback

Fixed err

Regen wasm file

Fixed based on feedback

Adding wasm file

fix  lint err

* fix  based on feedback

* Update wasm files
2020-01-30 15:03:14 -08:00
Pengyuan Bian 90d5f8dde3 Add extra labels to sd logging (#2625)
* add extra fields into request info

* add logic to export extended request info

* add initial value

* update wasm file

* address comment

* fix
2020-01-16 09:46:52 -08:00
Pengyuan Bian 6f7d891b30 update envoy wasm sha (#2622) 2020-01-09 14:01:09 -08:00
Nupur Garg 072630f6f0 Add support for Access Log Policy Filter (#2602)
* Add Access Log Sampling Filter

Format

Make it just mvp ready

Format

Fix extra line

Fix go-lint errors

Fix fmt

change to using absl::Time and absl::Duration instead of long int for time calculations

Fixed after feedback

Fix lint error

* Debug test failure in circleci

* Debug test failure in circleci

* Debug test failure in circleci

* Debug test failure in circleci

* Add lock for the cache

* Debug circle ci

* Fix test
2020-01-06 18:26:28 -08:00
Pengyuan Bian 5a66e29530 Extract service name from request host and cluster name (#2548)
* support extracing service name from request host

* service_name:port

* s/namespace_pos/name_pos

* update

* lint

* add initialization to service auth policy

* format

* use absl time

* update test

* address comment
2019-12-14 00:24:47 -08:00
Pengyuan Bian 821f2846f2 Increase OC interval to 60s (#2601) (#2604)
* add configurable metric export interval

* format

* update env variable
2019-12-13 12:34:22 -08:00
Pengyuan Bian d2a7d319a5 Update envoy wasm sha (#2579)
* update envoy-wasm sha

* fix abi
2019-11-21 01:39:23 -08:00
Lizan Zhou 73b18cae6f Update envoy-wasm sha (#2559)
* Update envoy-wasm sha

Signed-off-by: Lizan Zhou <lizan@tetrate.io>

* fix due to ABI change

* remove compile db

Signed-off-by: Lizan Zhou <lizan@tetrate.io>

* fix for envoyproxy/envoy-wasm#216

Signed-off-by: Lizan Zhou <lizan@tetrate.io>

* fix ci

* roll forward

Signed-off-by: Lizan Zhou <lizan@tetrate.io>
2019-11-15 10:43:46 -08:00
mandarjog 0ace1cc059 v2: Add ability to control host header fallback (#2546)
* WIP

* WIP2

* add test

* WIP5

* WIP7

* Update testdata

* fix go error

* Use dynamic client port
2019-11-10 21:07:53 -08:00