* Update istio/istio ref to latest master
* Move to latest before testing.
* Update release in Makefile
* Fix some tests
* Update to latest isti.io/istio again
* Update to latest istio.io/istio
* Update to latest istio.io/istio
* Update ref to latest master
* Fix instioctl-analyze
* Add back @howardjohn commit I inadvertantly deleted
* Fix lint
* Pick up new stio: fix empty iop read from stdin for operator
* go mod tidy without itermediate go gets
* Update to current stats-filter's
* Needed another 'make gen'
* More go.* changes for changes in ref'd istio/istio
* Update istio to 1.9.0-beta.0
Overview of the changes:
- Adding ability to verify that expected output occurs a number of times consecutively. This is needed for https://github.com/istio/istio.io/pull/8402.
- Moving snapshot checking logic to Go code so that it can be separated out into separate test steps, which are timed and contain their own output directories/files. This makes the code cleaner and also makes the snapshot logic more transparent.
- Updating debug.sh to use newer bash syntax that allows it to dynamically select a free file descriptor. Without this, I was seeing all commands echoed to my console in goland.
This required some other changes WRT verification:
- Change __cmp_like to allow for not accepting <pending> for an IP address.
- Change __verify_with_retry to use a timeout rathan than number of retries. This is a more intuitive interface and aligns with the way we do retries in istio/istio. I also got rid of exponential backoff and allow both the timeout and delay between retries to be configured.
This was noticed when investigating why errors are not causing the test to fail. It's not clear if this is related to the problem, but the existing logic was definitely atypical.
* Try longer timeout for tasks/security/authorization/authz-ingress/test.sh
* Go back to old 5m timeout and add failure on timeout back in
* Test with individual wait_for_istio call updates
* Revert to simply changing timeout (but keep duration in output)
* Fix call
* Comment out istioctl wait call
* Add TODO remark
* Fix lint error
* Add temporary sleep until _wait_for_istio is re-enabled
* Add wait for sleep deployment to complete
* test framework enhancement for multicluster doc test
* simplify multicluster test framework with setup=multicluster directive
* restore documentation comment
* Update test reference
* Test framework changes
* Another required change
* Update Tag to 1.8
* Pick istio/istio commit that actually exists
* Disable ISTIO_META_DNS_CAPTURE
* Add --skip-confirmation to istioctl installl commands
* Increase test timeout. First pass at fixes.
* Update to later istio/istio that fixes DNS and minor fixes
* test fixes
* Pick up go.mod `replace` changes from #8118
* Fix istioctl-analayze and mirror
* Fix mtls-migration test
* Update istio to include commit to fix egress
* Re-enable verify with fix
* Update istio/istio ref for egress fix
* Fix tasks/security/authorization/authz-td-migration - remove ns
* Shorten wait timeout so tests complete in under an hr
* Let tests continue after wait timeout
* Fix --skip-confirmation to -y and use yes | in tests
* revert yes | to echo y |
* Additional echo y fix
* Code review comments
* Change verify from same to contains as k8s 1.19 has extra warning lines.
* automated test - insecure remote access to telemetry addons
* Change profile to demo and add snippets to clean up DestinationRule (besides Gateway and VirtualService)
* automated doc test = secure remote access to telemetry addons through gateway
* Automated test for secure access of common telemetry addons and factor out common snippets to test_utils.sh
* moved test_utils to tests/util/addons.sh so that it can be reused across tests
* refactor addons with a more flexible deploy/undeploy function and fix naming of functions to access addons
* fix cleanup after tests
* fix sourcing
* remove verification for istio config deployment
* don't assume the existence of load-balancer in CI environment, but use _set_ingress_environment_variables instead of snippet in the documentation page
* add verification steps back
* fix output snippets and add verifications
* automated test for viewing traces from zipkin dashboard (expectation is that traces are seen correctly)
* fix shellcheck linting errors
* remove verification of traces and just check if zipkin dashboard is accessible through port-forward
* Modify snippet generation logic to take {{< boilerplate >}} into account
* snippets for boilerplate
* fix shellcheck (linting) issues
* snippets generated with new logic
* automated test - check if zipkin dashboard is accessible by port-forwarding (as we cannot verify screenshot and the trace is already verified in istio/istio integration test
* clean up ZIPKIN_URL comment
* Address code review comments
1. Simplify boilerplate snippet filenames (remove .md_snips)
2. Sourced filenames should be in double quote(") instead of single quote(')
* fix gencheck_istio flake
* do not generate or include boilerplate snippets if there is not {{<text>}} in them
* fix linting errors in snip.py
* fix bug - filter out boilerplates without snippets very early
* Update istio version for testing
* Framework changes in base istio required changes
* Work on whitelist
* Remove whitelist and move to a new feature
* Another istio.io/istio update
* Another framework change - remove test suite name
* add an example task to test
* main test function: save progress
* a working example: routing request
* improve log info and error handling
* introduce makefile
* run each test as a subtest; remove common setup from test.sh
* add another test.sh: fault-injection
* improve error handling
* check test environment
* add two more test.sh files
* fix make command for istio setup
* update two test.sh files from upstream
* add comments and update README.md
* update test.sh files from upstream
* support multiple test names
* update README
* update README.md for new framework
* remove documentation of migration steps
* undo format changes
* change separation line to '# @cleanup'
* move go code and makefile from content/ to tests/
* change package name
* make for loop more readable
* change the set of auto-sourced scripts
* add docs for all functions
* approach to deal with folders with the same name
* minor fixes to ensure everything still runs
* fix make gen error
* add a TIMEOUT argument
* make sure util/debug.sh works with new framework
* make lint-go happy
* [BIG CHANGE] allow different istio setup configs
* make linters happy
* make linters happier
* changed wording and function orders
* make error return as the 2nd argument
* add TODOs
* Update content/en/docs/tasks/traffic-management/traffic-shifting/test.sh
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* Update tests/README.md
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* Update tests/README.md
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* Update tests/README.md
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* Update tests/README.md
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* Update tests/README.md
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* Update tests/README.md
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* only test english docs
* Update tests/README.md
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* allow test.sh as suffix
* move adding setup configs to tests/setup
* recommend full paths
* Update tests/README.md
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* require full test paths
* converting old tests to new tests: traffic-management and misc
* converting old tests to new tests: security
* remove old tests
* Update content/en/docs/tasks/security/cert-management/dns-cert/test.sh
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* simplify setup configs
* Update content/en/docs/tasks/security/authentication/authn-policy/test.sh
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* Update content/en/docs/tasks/security/authentication/mtls-migration/test.sh
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* Update content/en/docs/tasks/security/authorization/authz-http/test.sh
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* do not let istioctl prompt y/n
* Update content/en/docs/tasks/traffic-management/ingress/ingress-sni-passthrough/test.sh
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* Update content/en/docs/tasks/traffic-management/ingress/secure-ingress/test.sh
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* Update content/en/docs/tasks/security/cert-management/plugin-ca-cert/test.sh
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* simplify stuff
* rename dns-cert test.sh to test_broken.sh
* fix dns-cert doc and test
* remove egress=disabled
* fix test
* Update content/en/docs/tasks/observability/logs/access-log/test.sh
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* Update content/en/docs/tasks/security/authentication/authn-policy/test.sh
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* add an example task to test
* main test function: save progress
* a working example: routing request
* improve log info and error handling
* introduce makefile
* run each test as a subtest; remove common setup from test.sh
* add another test.sh: fault-injection
* improve error handling
* check test environment
* add two more test.sh files
* fix make command for istio setup
* update two test.sh files from upstream
* add comments and update README.md
* update test.sh files from upstream
* support multiple test names
* update README
* update README.md for new framework
* remove documentation of migration steps
* undo format changes
* change separation line to '# @cleanup'
* move go code and makefile from content/ to tests/
* change package name
* make for loop more readable
* change the set of auto-sourced scripts
* add docs for all functions
* approach to deal with folders with the same name
* minor fixes to ensure everything still runs
* fix make gen error
* add a TIMEOUT argument
* make sure util/debug.sh works with new framework
* make lint-go happy
* [BIG CHANGE] allow different istio setup configs
* make linters happy
* make linters happier
* changed wording and function orders
* make error return as the 2nd argument
* add TODOs
* Update content/en/docs/tasks/traffic-management/traffic-shifting/test.sh
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* Update tests/README.md
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* Update tests/README.md
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* Update tests/README.md
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* Update tests/README.md
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* Update tests/README.md
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* Update tests/README.md
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* only test english docs
* Update tests/README.md
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* allow test.sh as suffix
* move adding setup configs to tests/setup
* recommend full paths
* Update tests/README.md
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* require full test paths
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* Update istio sha for tests to current `master` from `release-1.6`
* Update to latest `master` sha
* Change ISTIO_IMAGE_VERSION to match master
* Increase retries for flakiness
* Pick up istio release with egress fix
* Fix for flakiness
* authn-policy: Point users to Istio installation guide.
Have users install Istio through the installation guide instead of
explicitly instructing them to run "istioctl manifest apply" in the
task. This will make it easier to automate the task later.
* authn-policy: Make steps and expected response clear for testing.
* Add authn-policy user guide test.
When tests are run, shell script debugging is enabled and written to a
file called "<script_name>_debug.txt" in the same directory as the
"<script_name>_output.txt" file.
* verify.sh: Don't exit on error in _run_and_verify_*() funcs.
Most tests include "set -e", which causes the script to exit if a
statement returns a non-true return value. In some cases, the command
may exit with a non-true return value, but we want to retry the command
later. We want to temporarily disable that "errexit" behavior.
* verify.sh: Clarify __verify_with_retry() comment and variable name.
* snip.py: Replace github file token with release-specific URL.
* verify.sh: Show the expected output as well as the actual output.
* snip.py: Update the githubfile regex to not include email addresses.
When generating snip scripts, pairs of "@" signs indicate a link to
GitHub repo content. However, JWT attribute values contained pairs of
email addresses such as:
`testing@secure.istio.io/testing@secure.istio.io`
which would be treated as an email address and mangled. This commit
rewrites the regex to not match on email addresses.
* Add authz-jwt user guide test.
* Update test framework to use 1.6.0-beta.0
* Go back to copies from env var
* Add more test targets, fix mtls test (new PA added)
* Update to use Istio SHA from go.mod (convert to long SHA)
* Try and remove TAG from prow
* Debug TAG not being set
* Fix paralization issue
* Remove some extra output
* Review comments
Reverted the actual mirror test script, because mirror test seems to have some subtle failure when running with what seems to be the exact same commands via snips. Will investigate further in followup PR. Merging this one to get the generator changes.
* Add test for Request Timeouts
* Update test to extract snippets
* Apply suggestions from code review
Co-Authored-By: Frank Budinsky <frankb@ca.ibm.com>
* Fix lint error
* Use code snippets from bookinfo snips
Co-Authored-By: Frank Budinsky <frankb@ca.ibm.com>
* Remove sleep pods
* remove deprecated $snippet and use use verify func
Co-Authored-By: Frank Budinsky <frankb@ca.ibm.com>
* Skip test failure
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* User guide tests for DNS certificate management
- Add user guide tests for DNS certificate management
- Remove user guide's dependency on jq
* Use _verify_contains function
* User guide tests and remove manual steps for plugging in CA cert
- Add user guide tests for plugging in CA cert
- Remove the manual steps in the user guide of plugging in CA cert
to make it easier for an user to try the guide.
* Fix SC2046 lint error and the trafficmanagement test errors
* Update doc test README
* add sh
* tweaks
* formatting
* format
* fix comment
* wording
* convert mtls migration task
* Update tests/README.md
Co-Authored-By: Eric Van Norman <ericvn@us.ibm.com>
* Update tests/README.md
Co-Authored-By: Eric Van Norman <ericvn@us.ibm.com>
Co-authored-by: Eric Van Norman <ericvn@us.ibm.com>
The `TestCircuitBreaker` test is very flaky. It relies on examining
the percentage of requests that succeed or fail to a service with
circuit breaking configured, and ensuring the numbers fall within a
given range. Unfortunately, the percentages tend to vary widely from
run to run, so the test fails as often as it passes.
This simplifies the test by only verifying that both successes and
failures occur instead of checking the percentage of each.
Most of the tasks in the docs assume the installation is using the
demo profile, but the tests install Istio with the default profile.
At least one instance of an issue caused by this can be found in
istio/istio#21556. This changes the example tests to run against an
installation using the demo profile.
Fixes: https://github.com/istio/istio/issues/21634
The mTLS migration task in the docs was updated to use
PeerAuthentication instead of Policy and DestinationRules, but the
tests were never updated. This updates the test script to be inline
with the docs.
The `TestTCPTrafficShifting` test script relies on fetching the port
named "tcp" on the istio-ingressgateway service, but that port is not
present in the default installation profile -- only in the demo
profile. This patches the service to add the port in the setup
script, and also fixes the usage of `nc` in the test script.
* (#6586) move istio.io test for circuit breaking to istio.io repository
* (#6586) add verification for output metrics
* (#6586) widened bounds for both before and after tripping and use ruby for evaluation
* (#6586) fix istioio package import
Eric Van Norman
John Howard
Frank Budinsky
Nathan Mittler
Suchith J N
Mitch Connors
Hongyi Zhang
Hongyi Zhang
Iris
Justin Pettit
Lin Sun
Oliver Liu
lei-tang
Shamsher Ansari
Brad Ison
Brad Ison
Istio Automation
Daniel Grimm