istio.io/tests/README.md

Testing istio.io Content

This folder contains tests for the content on istio.io. More specifically, these tests confirm that the example and task documents, which contain instructions in the form of bash commands and expected output, are working as documented.

Generated bash scripts, containing the set of commands and expected output for corresponding istio.io markdown files, are used by test programs to invoke the commands and verify the output. This means that we extract and test the exact same commands that are published in the documents.

These tests use the framework defined in the istioio package, which is a thin wrapper around the Istio test framework.

Test Authoring Overview

To write an istio.io test, follow these steps:

1. Add a field test: true to the metadata at the top of the index.md file to be tested. This field is used to indicate that the markdown file will be tested and therefore requires a generated bash script containing the commands described in the document.

2. Run make snips to generate the bash script. After the command completes, you should see a new file, snips.sh, next to the index.md file that you modified in the previous step.

   Each bash command in index.md (i.e., {{< text bash >}} code block) will produce a bash function in snips.sh containing the same command(s) as in the document. Other types of code blocks, e.g., {{< text yaml >}}, will produce a bash variable containing the block content.

   By default, the bash function or variable will be named snip_<section>_<code block number>. For example, the first {{< text bash >}} code block in a section titled ## Apply weight-based routing will generate a bash function named snip_apply_weightbased_routing_1().

   You can override the default name by adding snip_id=<some name> to the corresponding text block attributes. For example {{< text syntax=bash snip_id=config_all_v1 >}} will generate snip_config_all_v1().

   If a bash code block contains both commands and output, the snips.sh script will include both a bash function and a variable containing the expected output. The name of the variable will be the same as the function, only with _out appended.

3. Run make lint-fast to check for script errors.

   If there are any lint errors in the generated snip.sh file, it means that a command in the index.md file is not following bash best practices. Because we are extracting the commands from the markdown file into a script file, we get the added benefit of lint checking of the commands that appear in the docs.

   Fix the errors, if any, by updating the corresponding command in the index.md file and then regenerate the snips.

4. Pick an appropriate location under the tests/ directory and create a directory for your new test.

5. Add the following imports to your GoLang file:

   "istio.io/istio/pkg/test/framework"
   "istio.io/istio/pkg/test/framework/components/environment"
   "istio.io/istio/pkg/test/framework/components/istio"
   
   "istio.io/istio.io/pkg/test/istioio"
   

6. Create a function called TestMain, following the example below. This function sets up the Istio environment that the test uses. The Setup function accepts an optional function to customize the Istio environment deployed.

   func TestMain(m *testing.M) {
   framework.NewSuite("my-istioio-test", m).
       SetupOnEnv(environment.Kube, istio.Setup(&ist, nil)).
       RequireEnvironment(environment.Kube).
       Run()
   }
   

7. To create a test, you use istioio.NewBuilder to build a series of steps that will be run as part of the resulting test function:

   func TestCombinedMethods(t *testing.T) {
       framework.
           NewTest(t).
           Run(istioio.NewBuilder("tasks__security__my_task").
               Add(istioio.Script{
                   Input:         istioio.Path("myscript.sh"),
               },
               istioio.MultiPodWait("foo"),
               istioio.Script{
                   Input:         istioio.Path("myotherscript.sh"),
               }).Build())
   }
   

Running Shell Commands

Your test will include one or more test steps that run shell scripts that call the commands in the generated snips.sh file.

istioio.Script{
    Input:   istioio.Path("myscript.sh"),
}

Your script must include the snip.sh file for the document being tested. For example, a test for the traffic-shifting task will have the following line in the script:

source ${REPO_ROOT}/content/en/docs/tasks/traffic-management/traffic-shifting/snips.sh

Your test script can then invoke the commands by simply calling snip functions:

snip_config_50_v3 # Step 3: switch 50% traffic to v3

For commands that produce output that needs to be verified, capture the command output in a variable and compare it to the expected output. For example:

out=$(snip_set_up_the_cluster_3 2>&1)
_verify_same "$out" "$snip_set_up_the_cluster_3_out" "snip_set_up_the_cluster_3"

The framework includes the following built-in verify functions:

1. _verify_same out expected msg

   Verify that out is exactly the same as expected. Failure messages will include the specified msg.

2. _verify_contains out expected msg

   Verify that out contains the substring expected. Failure messages will include the specified msg.

3. _verify_like out expected msg

   Verify that out is "like" expected. Like implies:

   • Same number of lines

   • Same number of whitespace-seperated tokens per line

   • Tokens can only differ in the following ways:

     1. different elapsed time values (e.g., 30s is like 5m)
     2. different ip values (e.g., 172.21.0.1 is like 10.0.0.31)
     3. prefix match ending with a dash character (e.g., reviews-v1-12345... is like reviews-v1-67890...)

   This function is useful for comparing the output of commands that include some run-specific values in the output (e.g., kubectl get pods), or when whitespace in the output may be different.

Builder

The istioio.NewBuilder returns a istioio.Builder that is used to build an Istio test run function and has the following methods:

• Add: adds a step to the test.
• Defer: provides a step to be run after the test completes.
• Build: builds an Istio test run function.

Selecting Input

Many test steps require an Input which they obtain from an istioio.InputSelector:

type Input interface {
    InputSelector
    Name() string
    ReadAll() (string, error)
}

type InputSelector interface {
    SelectInput(Context) Input
}

Some common InputSelector implementations include:

• istioio.Inline: allows you to inline the content for the Input directly in the code.
• istioio.Path: reads in a file from the specified path.
• istioio.BookInfo: is like istioio.Path except that the value is assumed to be relative to the BookInfo source directory ($GOPATH/src/istio.io/istio/samples/bookinfo/platform/kube/).

An InputSelector provides an istioio.Context at runtime, which it can use to dynamically choose an Input. For example, we could choose a different file depending on whether or not the test is running on Minikube:

istioio.InputSelectorFunc(func(ctx istioio.Context) Input {
    if ctx.Env.Settings().Minikube {
        return istioio.Path("scripts/curl-httpbin-tls-gateway-minikube.sh")
    }
    return istioio.Path("scripts/curl-httpbin-tls-gateway-gke.sh")
})

The library also provides a utility that helps simplify this particular use case:

istioio.IfMinikube{
    Then: istioio.Path("scripts/curl-httpbin-tls-gateway-minikube.sh")
    Else: istioio.Path("scripts/curl-httpbin-tls-gateway-gke.sh")
}

Waiting for Pods to Start

You can create a test step that waits for one or more pods to start before continuing. For example, to wait for all pods in the "foo" namespace, you can do the following:

istioio.MultiPodWait("foo"),

Running the Tests: Make

You can execute all istio.io tests using make.

export KUBECONFIG=~/.kube/config
make test.kube.presubmit

Notes:

There is an issue with the TAG (#7081) so one needs to set TAG to latest to mimic the pipeline.

In the case of using kind clusters on the Mac, an extra env var is needed, ADDITIONAL_CONTAINER_OPTIONS="--network host". If one makes sure HUB is not set, then the command TEST_ENV=kind ADDITIONAL_CONTAINER_OPTIONS="--network host" make test.kube.presubmit has been successful.

Running Tests: go test

You can execute individual tests using Go test as shown below.

make init
export REPO_ROOT=$(git rev-parse --show-toplevel)
go test ./tests/... -p 1  --istio.test.env kube \
    --istio.test.ci --istio.test.work_dir <my_dir>

The value of my_dir will be the parent directory for your test output. Within my_dir, each test Main will create a directory containing a subdirectory for each test method. Each test method directory will contain a snippet.txt that was generated for that particular test.

Make sure to have the HUB and TAG environment variables set to the location of your Istio Docker images.

You can find the complete list of arguments on the test framework wiki page.