This required some other changes WRT verification:
- Change __cmp_like to allow for not accepting <pending> for an IP address.
- Change __verify_with_retry to use a timeout rathan than number of retries. This is a more intuitive interface and aligns with the way we do retries in istio/istio. I also got rid of exponential backoff and allow both the timeout and delay between retries to be configured.
This was noticed when investigating why errors are not causing the test to fail. It's not clear if this is related to the problem, but the existing logic was definitely atypical.
* Try longer timeout for tasks/security/authorization/authz-ingress/test.sh
* Go back to old 5m timeout and add failure on timeout back in
* Test with individual wait_for_istio call updates
* Revert to simply changing timeout (but keep duration in output)
* Fix call
* Comment out istioctl wait call
* Add TODO remark
* Fix lint error
* Add temporary sleep until _wait_for_istio is re-enabled
* Add wait for sleep deployment to complete
* test framework enhancement for multicluster doc test
* simplify multicluster test framework with setup=multicluster directive
* restore documentation comment
* Update test reference
* Test framework changes
* Another required change
* Update Tag to 1.8
* Pick istio/istio commit that actually exists
* Disable ISTIO_META_DNS_CAPTURE
* Add --skip-confirmation to istioctl installl commands
* Increase test timeout. First pass at fixes.
* Update to later istio/istio that fixes DNS and minor fixes
* test fixes
* Pick up go.mod `replace` changes from #8118
* Fix istioctl-analayze and mirror
* Fix mtls-migration test
* Update istio to include commit to fix egress
* Re-enable verify with fix
* Update istio/istio ref for egress fix
* Fix tasks/security/authorization/authz-td-migration - remove ns
* Shorten wait timeout so tests complete in under an hr
* Let tests continue after wait timeout
* Fix --skip-confirmation to -y and use yes | in tests
* revert yes | to echo y |
* Additional echo y fix
* Code review comments
* Change verify from same to contains as k8s 1.19 has extra warning lines.
* automated test - insecure remote access to telemetry addons
* Change profile to demo and add snippets to clean up DestinationRule (besides Gateway and VirtualService)
* automated doc test = secure remote access to telemetry addons through gateway
* Automated test for secure access of common telemetry addons and factor out common snippets to test_utils.sh
* moved test_utils to tests/util/addons.sh so that it can be reused across tests
* refactor addons with a more flexible deploy/undeploy function and fix naming of functions to access addons
* fix cleanup after tests
* fix sourcing
* remove verification for istio config deployment
* don't assume the existence of load-balancer in CI environment, but use _set_ingress_environment_variables instead of snippet in the documentation page
* add verification steps back
* fix output snippets and add verifications
* automated test for viewing traces from zipkin dashboard (expectation is that traces are seen correctly)
* fix shellcheck linting errors
* remove verification of traces and just check if zipkin dashboard is accessible through port-forward
* Modify snippet generation logic to take {{< boilerplate >}} into account
* snippets for boilerplate
* fix shellcheck (linting) issues
* snippets generated with new logic
* automated test - check if zipkin dashboard is accessible by port-forwarding (as we cannot verify screenshot and the trace is already verified in istio/istio integration test
* clean up ZIPKIN_URL comment
* Address code review comments
1. Simplify boilerplate snippet filenames (remove .md_snips)
2. Sourced filenames should be in double quote(") instead of single quote(')
* fix gencheck_istio flake
* do not generate or include boilerplate snippets if there is not {{<text>}} in them
* fix linting errors in snip.py
* fix bug - filter out boilerplates without snippets very early
* Update istio version for testing
* Framework changes in base istio required changes
* Work on whitelist
* Remove whitelist and move to a new feature
* Another istio.io/istio update
* Another framework change - remove test suite name
* add an example task to test
* main test function: save progress
* a working example: routing request
* improve log info and error handling
* introduce makefile
* run each test as a subtest; remove common setup from test.sh
* add another test.sh: fault-injection
* improve error handling
* check test environment
* add two more test.sh files
* fix make command for istio setup
* update two test.sh files from upstream
* add comments and update README.md
* update test.sh files from upstream
* support multiple test names
* update README
* update README.md for new framework
* remove documentation of migration steps
* undo format changes
* change separation line to '# @cleanup'
* move go code and makefile from content/ to tests/
* change package name
* make for loop more readable
* change the set of auto-sourced scripts
* add docs for all functions
* approach to deal with folders with the same name
* minor fixes to ensure everything still runs
* fix make gen error
* add a TIMEOUT argument
* make sure util/debug.sh works with new framework
* make lint-go happy
* [BIG CHANGE] allow different istio setup configs
* make linters happy
* make linters happier
* changed wording and function orders
* make error return as the 2nd argument
* add TODOs
* Update content/en/docs/tasks/traffic-management/traffic-shifting/test.sh
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* Update tests/README.md
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* Update tests/README.md
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* Update tests/README.md
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* Update tests/README.md
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* Update tests/README.md
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* Update tests/README.md
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* only test english docs
* Update tests/README.md
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* allow test.sh as suffix
* move adding setup configs to tests/setup
* recommend full paths
* Update tests/README.md
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* require full test paths
* converting old tests to new tests: traffic-management and misc
* converting old tests to new tests: security
* remove old tests
* Update content/en/docs/tasks/security/cert-management/dns-cert/test.sh
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* simplify setup configs
* Update content/en/docs/tasks/security/authentication/authn-policy/test.sh
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* Update content/en/docs/tasks/security/authentication/mtls-migration/test.sh
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* Update content/en/docs/tasks/security/authorization/authz-http/test.sh
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* do not let istioctl prompt y/n
* Update content/en/docs/tasks/traffic-management/ingress/ingress-sni-passthrough/test.sh
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* Update content/en/docs/tasks/traffic-management/ingress/secure-ingress/test.sh
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* Update content/en/docs/tasks/security/cert-management/plugin-ca-cert/test.sh
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* simplify stuff
* rename dns-cert test.sh to test_broken.sh
* fix dns-cert doc and test
* remove egress=disabled
* fix test
* Update content/en/docs/tasks/observability/logs/access-log/test.sh
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* Update content/en/docs/tasks/security/authentication/authn-policy/test.sh
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* add an example task to test
* main test function: save progress
* a working example: routing request
* improve log info and error handling
* introduce makefile
* run each test as a subtest; remove common setup from test.sh
* add another test.sh: fault-injection
* improve error handling
* check test environment
* add two more test.sh files
* fix make command for istio setup
* update two test.sh files from upstream
* add comments and update README.md
* update test.sh files from upstream
* support multiple test names
* update README
* update README.md for new framework
* remove documentation of migration steps
* undo format changes
* change separation line to '# @cleanup'
* move go code and makefile from content/ to tests/
* change package name
* make for loop more readable
* change the set of auto-sourced scripts
* add docs for all functions
* approach to deal with folders with the same name
* minor fixes to ensure everything still runs
* fix make gen error
* add a TIMEOUT argument
* make sure util/debug.sh works with new framework
* make lint-go happy
* [BIG CHANGE] allow different istio setup configs
* make linters happy
* make linters happier
* changed wording and function orders
* make error return as the 2nd argument
* add TODOs
* Update content/en/docs/tasks/traffic-management/traffic-shifting/test.sh
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* Update tests/README.md
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* Update tests/README.md
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* Update tests/README.md
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* Update tests/README.md
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* Update tests/README.md
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* Update tests/README.md
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* only test english docs
* Update tests/README.md
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* allow test.sh as suffix
* move adding setup configs to tests/setup
* recommend full paths
* Update tests/README.md
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* require full test paths
Co-authored-by: Frank Budinsky <frankb@ca.ibm.com>
* Update istio sha for tests to current `master` from `release-1.6`
* Update to latest `master` sha
* Change ISTIO_IMAGE_VERSION to match master
* Increase retries for flakiness
* Pick up istio release with egress fix
* Fix for flakiness
* authn-policy: Point users to Istio installation guide.
Have users install Istio through the installation guide instead of
explicitly instructing them to run "istioctl manifest apply" in the
task. This will make it easier to automate the task later.
* authn-policy: Make steps and expected response clear for testing.
* Add authn-policy user guide test.
When tests are run, shell script debugging is enabled and written to a
file called "<script_name>_debug.txt" in the same directory as the
"<script_name>_output.txt" file.
Nathan Mittler
Eric Van Norman
Suchith J N
Frank Budinsky
Mitch Connors
Hongyi Zhang
Hongyi Zhang
Iris
Justin Pettit
Lin Sun
John Howard