Adding Version 2.0.0-Beta8 Docs. (#41)
Signed-off-by: Jonsy13 <vedant.shrotria@chaosnative.com>
This commit is contained in:
VEDANT SHROTRIA
GitHub
parent
754786cb13
commit
a6cbe1166f
|
|
@ -15,7 +15,7 @@ helm uninstall litmuschaos --namespace litmus
|
|||
kubectl delete ns litmus
|
||||
```
|
||||
|
||||
----
|
||||
----
|
||||
|
||||
### **Uninstallation of Litmus Control Plane using kubectl**
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,196 @@
|
|||
---
|
||||
id: agent-install
|
||||
title: Litmus Chaos Agent Install
|
||||
sidebar_label: Chaos Agent
|
||||
---
|
||||
|
||||
In Litmus the Agents can be classified as two types
|
||||
|
||||
- Self Agent
|
||||
- External Agent
|
||||
|
||||
As part of Litmus installation by default, a self cluster would be registered as Agent in the Portal. From Portal you induce chaos into self cluster and observe the results from the Portal.
|
||||
|
||||
As you are aware by now, Portal is a Cross Cloud Chaos Control plane. That is you can connect multiple external kubernetes agents to this portal. Once connected you can manage the chaos from the Portal that is you can induce chaos into this agent from the Portal and observe the results from the Portal. Using the command line utility _litmusctl_ you can connect the external agents to the Portal.
|
||||
|
||||
# Litmusctl
|
||||
|
||||
Litmusctl is a command line interface to manage LitmusPortal services.
|
||||
|
||||
## Requirements
|
||||
|
||||
The litmusctl CLI requires the following things:
|
||||
|
||||
- Kubeconfig - litmusctl needs the kubeconfig of the k8s cluster where we need to connect litmus agents. The CLI currently uses the default path of kubeconfig i.e. `~/.kube/config`.
|
||||
|
||||
## Installation
|
||||
|
||||
To install the latest version of litmusctl follow the below steps:
|
||||
|
||||
- Download the latest litmusctl(master) binary from:
|
||||
|
||||
| Platforms | Download Link |
|
||||
| ------------------------------ | ------------------------------------------------------------------------------------------------------- |
|
||||
| litmusctl-darwin-386 (MacOS) | [Click here](https://litmusctl-bucket.s3-eu-west-1.amazonaws.com/litmusctl-darwin-386-master.tar.gz) |
|
||||
| litmusctl-darwin-amd64 (MacOS) | [Click here](https://litmusctl-bucket.s3-eu-west-1.amazonaws.com/litmusctl-darwin-amd64-master.tar.gz) |
|
||||
| litmusctl-linux-386 | [Click here](https://litmusctl-bucket.s3-eu-west-1.amazonaws.com/litmusctl-linux-386-master.tar.gz) |
|
||||
| litmusctl-linux-amd64 | [Click here](https://litmusctl-bucket.s3-eu-west-1.amazonaws.com/litmusctl-linux-amd64-master.tar.gz) |
|
||||
| litmusctl-linux-arm | [Click here](https://litmusctl-bucket.s3-eu-west-1.amazonaws.com/litmusctl-linux-arm-master.tar.gz) |
|
||||
| litmusctl-linux-arm64 | [Click here](https://litmusctl-bucket.s3-eu-west-1.amazonaws.com/litmusctl-linux-arm64-master.tar.gz) |
|
||||
| litmusctl-windows-386 | [Click here](https://litmusctl-bucket.s3-eu-west-1.amazonaws.com/litmusctl-windows-386-master.tar.gz) |
|
||||
| litmusctl-windows-amd64 | [Click here](https://litmusctl-bucket.s3-eu-west-1.amazonaws.com/litmusctl-windows-amd64-master.tar.gz) |
|
||||
| litmusctl-windows-arm | [Click here](https://litmusctl-bucket.s3-eu-west-1.amazonaws.com/litmusctl-windows-arm-master.tar.gz) |
|
||||
|
||||
<br />
|
||||
|
||||
- Download the litmusctl(v0.2.0) binary from:
|
||||
|
||||
| Platforms | Download Link |
|
||||
| ------------------------------ | ------------------------------------------------------------------------------------------------------- |
|
||||
| litmusctl-darwin-386 (MacOS) | [Click here](https://litmusctl-bucket.s3-eu-west-1.amazonaws.com/litmusctl-darwin-386-v0.2.0.tar.gz) |
|
||||
| litmusctl-darwin-amd64 (MacOS) | [Click here](https://litmusctl-bucket.s3-eu-west-1.amazonaws.com/litmusctl-darwin-amd64-v0.2.0.tar.gz) |
|
||||
| litmusctl-linux-386 | [Click here](https://litmusctl-bucket.s3-eu-west-1.amazonaws.com/litmusctl-linux-386-v0.2.0.tar.gz) |
|
||||
| litmusctl-linux-amd64 | [Click here](https://litmusctl-bucket.s3-eu-west-1.amazonaws.com/litmusctl-linux-amd64-v0.2.0.tar.gz) |
|
||||
| litmusctl-linux-arm | [Click here](https://litmusctl-bucket.s3-eu-west-1.amazonaws.com/litmusctl-linux-arm-v0.2.0.tar.gz) |
|
||||
| litmusctl-linux-arm64 | [Click here](https://litmusctl-bucket.s3-eu-west-1.amazonaws.com/litmusctl-linux-arm64-v0.2.0.tar.gz) |
|
||||
| litmusctl-windows-386 | [Click here](https://litmusctl-bucket.s3-eu-west-1.amazonaws.com/litmusctl-windows-386-v0.2.0.tar.gz) |
|
||||
| litmusctl-windows-amd64 | [Click here](https://litmusctl-bucket.s3-eu-west-1.amazonaws.com/litmusctl-windows-amd64-v0.2.0.tar.gz) |
|
||||
| litmusctl-windows-arm | [Click here](https://litmusctl-bucket.s3-eu-west-1.amazonaws.com/litmusctl-windows-arm-v0.2.0.tar.gz) |
|
||||
|
||||
<br />
|
||||
|
||||
- Extract the binary
|
||||
|
||||
```shell
|
||||
$ tar -zxvf litmusctl-<OS>-<ARCH>-<VERSION>.tar.gz
|
||||
```
|
||||
|
||||
- Provide necessary permissions
|
||||
|
||||
```shell
|
||||
$ chmod +x litmusctl
|
||||
```
|
||||
|
||||
- Move the litmusctl binary to /usr/local/bin/litmusctl
|
||||
|
||||
```shell
|
||||
$ sudo mv litmusctl /usr/local/bin/litmusctl
|
||||
```
|
||||
|
||||
## Basic Commands
|
||||
|
||||
litmusctl CLI command has the following structure:
|
||||
|
||||
```shell
|
||||
$ litmusctl <command> <subcommand> <subcommand> [options and parameters]
|
||||
```
|
||||
|
||||
To get the version of the litmusctl CLI:
|
||||
|
||||
```shell
|
||||
$ litmusctl version
|
||||
```
|
||||
|
||||
### Connecting an agent
|
||||
|
||||
To connect Litmus Chaos agent:
|
||||
|
||||
```shell
|
||||
$ litmusctl agent connect
|
||||
```
|
||||
|
||||
Next, you need to enter LitmusPortal details to login into your LitmusPortal account. Fields to be filled in:
|
||||
|
||||
**LitmusPortal UI URL:** Enter the URL used to access the Litmus Portal UI.
|
||||
Example, http://172.17.0.2:31696/
|
||||
|
||||
**Username:** Enter your LitmusPortal username.
|
||||
**Password:** Enter your LitmusPortal password.
|
||||
|
||||
```shell
|
||||
🔥 Connecting LitmusChaos agent
|
||||
|
||||
📶 Please enter LitmusChaos details --
|
||||
👉 Host URL where litmus is installed: http://172.17.0.2:31696/
|
||||
🤔 Username [admin]: admin
|
||||
🙈 Password:
|
||||
✅ Login Successful!
|
||||
```
|
||||
|
||||
Upon successful login, there will be a list of exiting projects displayed on the terminal. Select the desired project by entering the sequence number indicated against it.
|
||||
|
||||
```shell
|
||||
✨ Projects List:
|
||||
1. abc
|
||||
|
||||
🔎 Select Project: 1
|
||||
```
|
||||
|
||||
Next, select the installation mode. In case the selected mode was a Cluster there will be a prerequisites check to verify ClusterRole and ClusterRoleBinding.
|
||||
|
||||
```shell
|
||||
🔌 Installation Modes:
|
||||
1. Cluster
|
||||
2. Namespace
|
||||
|
||||
👉 Select Mode [cluster]: 1
|
||||
|
||||
🏃 Running prerequisites check....
|
||||
🔑 clusterrole - ✅
|
||||
🔑 clusterrolebinding - ✅
|
||||
|
||||
🌟 Sufficient permissions. Connecting Agent
|
||||
```
|
||||
|
||||
Next, enter the details of the new agent.
|
||||
|
||||
Fields to filled in:
|
||||
**Agent Name:** Enter the name for the new agent.
|
||||
|
||||
**Agent Description:** Fill in details about the agent.
|
||||
|
||||
**Platform Name:** Enter the platform name on which this agent is hosted. For example, AWS, GCP, Rancher etc.
|
||||
|
||||
**Enter the namespace:** You can either enter an existing namespace or enter a new namespace. In cases where the namespace does not exist, LitmusPortal creates it for you.
|
||||
|
||||
**Enter service account:** Enter a name for your service account.
|
||||
|
||||
```shell
|
||||
🔗 Enter the details of the agent ----
|
||||
🤷 Agent Name: my-agent
|
||||
📘 Agent Description: This is a new agent.
|
||||
📦 Platform List
|
||||
1. AWS
|
||||
2. GKE
|
||||
3. Openshift
|
||||
4. Rancher
|
||||
5. Others
|
||||
🔎 Select Platform [Others]: 5
|
||||
📁 Enter the namespace (new or existing) [litmus]: litmus
|
||||
🔑 Enter service account [litmus]: litmus
|
||||
```
|
||||
|
||||
Once, all these steps are implemented you will be able to see a summary of all the entered fields.
|
||||
After verification of these details, you can proceed with the connection of the agent by entering Y. The process of connection might take up to a few seconds.
|
||||
|
||||
```shell
|
||||
📌 Summary --------------------------
|
||||
|
||||
Agent Name: my-agent
|
||||
Agent Description: This is a new agent.
|
||||
Platform Name: Others
|
||||
Namespace: litmus
|
||||
Service Account: litmus
|
||||
Installation Mode: cluster
|
||||
|
||||
-------------------------------------
|
||||
|
||||
🤷 Do you want to continue with the above details? [Y/N]: Y
|
||||
|
||||
💡 Connecting agent to Litmus Portal.
|
||||
🏃 Agents running!!
|
||||
🚀 Agent Connection Successful!! 🎉
|
||||
👉 Litmus agents can be accessed here: http://172.17.0.2:31696/targets
|
||||
```
|
||||
|
||||
To verify, if the connection process was successful you can view the list of connected agents from the Targets section on your LitmusPortal and ensure that the connected agent is in Active State.
|
||||
|
|
@ -0,0 +1,26 @@
|
|||
---
|
||||
id: agent-uninstall
|
||||
title: Litmus Chaos Agent Uninstall
|
||||
sidebar_label: Chaos Agent
|
||||
---
|
||||
|
||||
As part of uninstalling agent components, we need to perform two steps
|
||||
1. Disconnect the Agent from Portal
|
||||
|
||||
- Login to Chaos Control Plane
|
||||
|
||||
- Click on Agent
|
||||
|
||||
- Click on the Disconnect icon on the required Chaos Agent which needs to be disconnected
|
||||
|
||||
- On the above operation it would remove the subscriber component from Chaos Agent and removes the connectivity between the Chaos Agent and Chaos Control Plane.
|
||||
|
||||
**Note** If the Chaos Agent is not reachable it would remove only the entry from the database of the Chaos control plane
|
||||
|
||||
2. Remove the Chaos Agent components from the system manually
|
||||
|
||||
|
||||
```bash
|
||||
kubectl delete chaosengine,chaosexperiments,chaosresults --all -A
|
||||
kubectl delete ns litmus
|
||||
```
|
||||
|
|
@ -0,0 +1,81 @@
|
|||
---
|
||||
id: architecture
|
||||
title: Litmus Architecture
|
||||
sidebar_label: Architecture
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
<img src={require('./assets/portal-arch.jpg').default} width="800" />
|
||||
|
||||
The above picture gives you a high-level architecture of the Litmus. At highlevel, Litmus components can be classified into two parts
|
||||
|
||||
1. Portal
|
||||
2. Agents
|
||||
|
||||
**Portal** is a set of Litmus components which act as Cross Cloud Chaos Control plane (WebUI) which is be used to orchestrate and observe the chaos workflows on Agents.
|
||||
|
||||
**Agent** is the set of Litmus components which induces Chaos using the chaos workflows on the K8S cluster component.
|
||||
|
||||
Typical user scenario, The user would install litmus. This would in-turn install Portal and Agent on the self cluster. Using the portal user can create/schedule new chaos workflows on the Agents and observe the results from here. User can also connect more clusters to the portal, and use the Portal as single window pane for cross cloud chaos management.
|
||||
|
||||
|
||||
|
||||
**Portal Components**
|
||||
|
||||
Portal has the following components
|
||||
|
||||
- Litmus WebUI
|
||||
|
||||
Litmus UI provides web user interface, where user can construct and observe the chaos workflow at ease. Also this act as cross cloud chaos control plane that is
|
||||
|
||||
- Litmus Server
|
||||
|
||||
Litmus Server act as middle ware which is use to handle API request from the user interface, store the config and results details into the DB. This also act as interface to communicate between the requests and scheduling the workflow to Agent.
|
||||
|
||||
- Litmus DB
|
||||
|
||||
Litmus DB act as config store for chaos workflows and its results.
|
||||
|
||||
**Agent components**
|
||||
|
||||
Agents has the following Litmus components
|
||||
|
||||
- Chaos Operator
|
||||
|
||||
Chaos-Operator watches for the ChaosEngine CR and executes the Chaos-Experiments mentioned in the CR. Chaos-Operator is namespace scoped. By default, it runs in `litmus` namespace. Once the experiment is completed, chaos-operator invokes chaos-exporter to export chaos metrics to a Prometheus database.
|
||||
|
||||
- CRDs
|
||||
|
||||
During installation, the following three CRDs are installed on the Kubernetes cluster.
|
||||
|
||||
```
|
||||
chaosexperiments.litmuschaos.io
|
||||
chaosengines.litmuschaos.io
|
||||
chaosresults.litmuschaos.io
|
||||
```
|
||||
|
||||
- Chaos Experiment
|
||||
|
||||
Chaos Experiment is a CR and are available as YAML files on [Chaos Hub](https://hub.litmuschaos.io/). For more details visit Chaos Hub [documentation](https://litmusdocs-beta.netlify.app/docs/chaoshub).
|
||||
|
||||
- Chaos Engine
|
||||
|
||||
ChaosEngine CR links application to experiments. User has to create ChaosEngine YAML by specifying the application label and experiments and create the CR. The CR is watched by Chaos-Operator and chaos-experiments are executed on a given application.
|
||||
|
||||
- Chaos Results
|
||||
|
||||
ChaosResult resource holds the results of a ChaosExperiment with a namespace scope. It is created or updated at runtime by the experiment itself. It holds important information like the ChaosEngine reference, Experiment State, Verdict of the experiment (on completion), salient application/result attributes. It is also a source for metrics collection. It is updated/patched with the status of the experiment run. It is not removed as part of the default cleanup procedures to allow for extended reference.
|
||||
|
||||
- Chaos Probes
|
||||
|
||||
Litmus probes are pluggable checks that can be defined within the ChaosEngine for any chaos experiment. The experiment pods execute these checks based on the mode they are defined in & factor their success as necessary conditions in determining the verdict of the experiment (along with the standard “in-built” checks).
|
||||
|
||||
- Chaos Exporter
|
||||
|
||||
Optionally metrics can be exported to a Prometheus database. Chaos-Exporter implements the Prometheus metrics endpoint.
|
||||
|
||||
- Subscriber
|
||||
|
||||
Subscriber is the component used in Agent side which interact with Litmus Server component to get the details of Chaos workflow and send the results back.
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 33 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 134 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 15 KiB |
|
|
@ -0,0 +1,429 @@
|
|||
<?xml version="1.0"?>
|
||||
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" width="1349.33333328" height="862.4" font-family="Consolas, Menlo, 'Bitstream Vera Sans Mono', monospace, 'Powerline Symbols'" font-size="14px">
|
||||
<style>
|
||||
<!-- asciinema theme -->
|
||||
|
||||
.default-text-fill {fill: #cccccc}
|
||||
.default-bg-fill {fill: #121314}
|
||||
|
||||
.c-0 {fill: #000000}
|
||||
.c-1 {fill: #dd3c69}
|
||||
.c-2 {fill: #4ebf22}
|
||||
.c-3 {fill: #ddaf3c}
|
||||
.c-4 {fill: #26b0d7}
|
||||
.c-5 {fill: #b954e1}
|
||||
.c-6 {fill: #54e1b9}
|
||||
.c-7 {fill: #d9d9d9}
|
||||
.c-8 {fill: #4d4d4d}
|
||||
.c-9 {fill: #dd3c69}
|
||||
.c-10 {fill: #4ebf22}
|
||||
.c-11 {fill: #ddaf3c}
|
||||
.c-12 {fill: #26b0d7}
|
||||
.c-13 {fill: #b954e1}
|
||||
.c-14 {fill: #54e1b9}
|
||||
.c-15 {fill: #ffffff}
|
||||
.c-8, .c-9, .c-10, .c-11, .c-12, .c-13, .c-14, .c-15 {font-weight: bold}
|
||||
<!-- 256 colors -->
|
||||
|
||||
.c-16 {fill: #000000}
|
||||
.c-17 {fill: #00005f}
|
||||
.c-18 {fill: #000087}
|
||||
.c-19 {fill: #0000af}
|
||||
.c-20 {fill: #0000d7}
|
||||
.c-21 {fill: #0000ff}
|
||||
.c-22 {fill: #005f00}
|
||||
.c-23 {fill: #005f5f}
|
||||
.c-24 {fill: #005f87}
|
||||
.c-25 {fill: #005faf}
|
||||
.c-26 {fill: #005fd7}
|
||||
.c-27 {fill: #005fff}
|
||||
.c-28 {fill: #008700}
|
||||
.c-29 {fill: #00875f}
|
||||
.c-30 {fill: #008787}
|
||||
.c-31 {fill: #0087af}
|
||||
.c-32 {fill: #0087d7}
|
||||
.c-33 {fill: #0087ff}
|
||||
.c-34 {fill: #00af00}
|
||||
.c-35 {fill: #00af5f}
|
||||
.c-36 {fill: #00af87}
|
||||
.c-37 {fill: #00afaf}
|
||||
.c-38 {fill: #00afd7}
|
||||
.c-39 {fill: #00afff}
|
||||
.c-40 {fill: #00d700}
|
||||
.c-41 {fill: #00d75f}
|
||||
.c-42 {fill: #00d787}
|
||||
.c-43 {fill: #00d7af}
|
||||
.c-44 {fill: #00d7d7}
|
||||
.c-45 {fill: #00d7ff}
|
||||
.c-46 {fill: #00ff00}
|
||||
.c-47 {fill: #00ff5f}
|
||||
.c-48 {fill: #00ff87}
|
||||
.c-49 {fill: #00ffaf}
|
||||
.c-50 {fill: #00ffd7}
|
||||
.c-51 {fill: #00ffff}
|
||||
.c-52 {fill: #5f0000}
|
||||
.c-53 {fill: #5f005f}
|
||||
.c-54 {fill: #5f0087}
|
||||
.c-55 {fill: #5f00af}
|
||||
.c-56 {fill: #5f00d7}
|
||||
.c-57 {fill: #5f00ff}
|
||||
.c-58 {fill: #5f5f00}
|
||||
.c-59 {fill: #5f5f5f}
|
||||
.c-60 {fill: #5f5f87}
|
||||
.c-61 {fill: #5f5faf}
|
||||
.c-62 {fill: #5f5fd7}
|
||||
.c-63 {fill: #5f5fff}
|
||||
.c-64 {fill: #5f8700}
|
||||
.c-65 {fill: #5f875f}
|
||||
.c-66 {fill: #5f8787}
|
||||
.c-67 {fill: #5f87af}
|
||||
.c-68 {fill: #5f87d7}
|
||||
.c-69 {fill: #5f87ff}
|
||||
.c-70 {fill: #5faf00}
|
||||
.c-71 {fill: #5faf5f}
|
||||
.c-72 {fill: #5faf87}
|
||||
.c-73 {fill: #5fafaf}
|
||||
.c-74 {fill: #5fafd7}
|
||||
.c-75 {fill: #5fafff}
|
||||
.c-76 {fill: #5fd700}
|
||||
.c-77 {fill: #5fd75f}
|
||||
.c-78 {fill: #5fd787}
|
||||
.c-79 {fill: #5fd7af}
|
||||
.c-80 {fill: #5fd7d7}
|
||||
.c-81 {fill: #5fd7ff}
|
||||
.c-82 {fill: #5fff00}
|
||||
.c-83 {fill: #5fff5f}
|
||||
.c-84 {fill: #5fff87}
|
||||
.c-85 {fill: #5fffaf}
|
||||
.c-86 {fill: #5fffd7}
|
||||
.c-87 {fill: #5fffff}
|
||||
.c-88 {fill: #870000}
|
||||
.c-89 {fill: #87005f}
|
||||
.c-90 {fill: #870087}
|
||||
.c-91 {fill: #8700af}
|
||||
.c-92 {fill: #8700d7}
|
||||
.c-93 {fill: #8700ff}
|
||||
.c-94 {fill: #875f00}
|
||||
.c-95 {fill: #875f5f}
|
||||
.c-96 {fill: #875f87}
|
||||
.c-97 {fill: #875faf}
|
||||
.c-98 {fill: #875fd7}
|
||||
.c-99 {fill: #875fff}
|
||||
.c-100 {fill: #878700}
|
||||
.c-101 {fill: #87875f}
|
||||
.c-102 {fill: #878787}
|
||||
.c-103 {fill: #8787af}
|
||||
.c-104 {fill: #8787d7}
|
||||
.c-105 {fill: #8787ff}
|
||||
.c-106 {fill: #87af00}
|
||||
.c-107 {fill: #87af5f}
|
||||
.c-108 {fill: #87af87}
|
||||
.c-109 {fill: #87afaf}
|
||||
.c-110 {fill: #87afd7}
|
||||
.c-111 {fill: #87afff}
|
||||
.c-112 {fill: #87d700}
|
||||
.c-113 {fill: #87d75f}
|
||||
.c-114 {fill: #87d787}
|
||||
.c-115 {fill: #87d7af}
|
||||
.c-116 {fill: #87d7d7}
|
||||
.c-117 {fill: #87d7ff}
|
||||
.c-118 {fill: #87ff00}
|
||||
.c-119 {fill: #87ff5f}
|
||||
.c-120 {fill: #87ff87}
|
||||
.c-121 {fill: #87ffaf}
|
||||
.c-122 {fill: #87ffd7}
|
||||
.c-123 {fill: #87ffff}
|
||||
.c-124 {fill: #af0000}
|
||||
.c-125 {fill: #af005f}
|
||||
.c-126 {fill: #af0087}
|
||||
.c-127 {fill: #af00af}
|
||||
.c-128 {fill: #af00d7}
|
||||
.c-129 {fill: #af00ff}
|
||||
.c-130 {fill: #af5f00}
|
||||
.c-131 {fill: #af5f5f}
|
||||
.c-132 {fill: #af5f87}
|
||||
.c-133 {fill: #af5faf}
|
||||
.c-134 {fill: #af5fd7}
|
||||
.c-135 {fill: #af5fff}
|
||||
.c-136 {fill: #af8700}
|
||||
.c-137 {fill: #af875f}
|
||||
.c-138 {fill: #af8787}
|
||||
.c-139 {fill: #af87af}
|
||||
.c-140 {fill: #af87d7}
|
||||
.c-141 {fill: #af87ff}
|
||||
.c-142 {fill: #afaf00}
|
||||
.c-143 {fill: #afaf5f}
|
||||
.c-144 {fill: #afaf87}
|
||||
.c-145 {fill: #afafaf}
|
||||
.c-146 {fill: #afafd7}
|
||||
.c-147 {fill: #afafff}
|
||||
.c-148 {fill: #afd700}
|
||||
.c-149 {fill: #afd75f}
|
||||
.c-150 {fill: #afd787}
|
||||
.c-151 {fill: #afd7af}
|
||||
.c-152 {fill: #afd7d7}
|
||||
.c-153 {fill: #afd7ff}
|
||||
.c-154 {fill: #afff00}
|
||||
.c-155 {fill: #afff5f}
|
||||
.c-156 {fill: #afff87}
|
||||
.c-157 {fill: #afffaf}
|
||||
.c-158 {fill: #afffd7}
|
||||
.c-159 {fill: #afffff}
|
||||
.c-160 {fill: #d70000}
|
||||
.c-161 {fill: #d7005f}
|
||||
.c-162 {fill: #d70087}
|
||||
.c-163 {fill: #d700af}
|
||||
.c-164 {fill: #d700d7}
|
||||
.c-165 {fill: #d700ff}
|
||||
.c-166 {fill: #d75f00}
|
||||
.c-167 {fill: #d75f5f}
|
||||
.c-168 {fill: #d75f87}
|
||||
.c-169 {fill: #d75faf}
|
||||
.c-170 {fill: #d75fd7}
|
||||
.c-171 {fill: #d75fff}
|
||||
.c-172 {fill: #d78700}
|
||||
.c-173 {fill: #d7875f}
|
||||
.c-174 {fill: #d78787}
|
||||
.c-175 {fill: #d787af}
|
||||
.c-176 {fill: #d787d7}
|
||||
.c-177 {fill: #d787ff}
|
||||
.c-178 {fill: #d7af00}
|
||||
.c-179 {fill: #d7af5f}
|
||||
.c-180 {fill: #d7af87}
|
||||
.c-181 {fill: #d7afaf}
|
||||
.c-182 {fill: #d7afd7}
|
||||
.c-183 {fill: #d7afff}
|
||||
.c-184 {fill: #d7d700}
|
||||
.c-185 {fill: #d7d75f}
|
||||
.c-186 {fill: #d7d787}
|
||||
.c-187 {fill: #d7d7af}
|
||||
.c-188 {fill: #d7d7d7}
|
||||
.c-189 {fill: #d7d7ff}
|
||||
.c-190 {fill: #d7ff00}
|
||||
.c-191 {fill: #d7ff5f}
|
||||
.c-192 {fill: #d7ff87}
|
||||
.c-193 {fill: #d7ffaf}
|
||||
.c-194 {fill: #d7ffd7}
|
||||
.c-195 {fill: #d7ffff}
|
||||
.c-196 {fill: #ff0000}
|
||||
.c-197 {fill: #ff005f}
|
||||
.c-198 {fill: #ff0087}
|
||||
.c-199 {fill: #ff00af}
|
||||
.c-200 {fill: #ff00d7}
|
||||
.c-201 {fill: #ff00ff}
|
||||
.c-202 {fill: #ff5f00}
|
||||
.c-203 {fill: #ff5f5f}
|
||||
.c-204 {fill: #ff5f87}
|
||||
.c-205 {fill: #ff5faf}
|
||||
.c-206 {fill: #ff5fd7}
|
||||
.c-207 {fill: #ff5fff}
|
||||
.c-208 {fill: #ff8700}
|
||||
.c-209 {fill: #ff875f}
|
||||
.c-210 {fill: #ff8787}
|
||||
.c-211 {fill: #ff87af}
|
||||
.c-212 {fill: #ff87d7}
|
||||
.c-213 {fill: #ff87ff}
|
||||
.c-214 {fill: #ffaf00}
|
||||
.c-215 {fill: #ffaf5f}
|
||||
.c-216 {fill: #ffaf87}
|
||||
.c-217 {fill: #ffafaf}
|
||||
.c-218 {fill: #ffafd7}
|
||||
.c-219 {fill: #ffafff}
|
||||
.c-220 {fill: #ffd700}
|
||||
.c-221 {fill: #ffd75f}
|
||||
.c-222 {fill: #ffd787}
|
||||
.c-223 {fill: #ffd7af}
|
||||
.c-224 {fill: #ffd7d7}
|
||||
.c-225 {fill: #ffd7ff}
|
||||
.c-226 {fill: #ffff00}
|
||||
.c-227 {fill: #ffff5f}
|
||||
.c-228 {fill: #ffff87}
|
||||
.c-229 {fill: #ffffaf}
|
||||
.c-230 {fill: #ffffd7}
|
||||
.c-231 {fill: #ffffff}
|
||||
.c-232 {fill: #080808}
|
||||
.c-233 {fill: #121212}
|
||||
.c-234 {fill: #1c1c1c}
|
||||
.c-235 {fill: #262626}
|
||||
.c-236 {fill: #303030}
|
||||
.c-237 {fill: #3a3a3a}
|
||||
.c-238 {fill: #444444}
|
||||
.c-239 {fill: #4e4e4e}
|
||||
.c-240 {fill: #585858}
|
||||
.c-241 {fill: #626262}
|
||||
.c-242 {fill: #6c6c6c}
|
||||
.c-243 {fill: #767676}
|
||||
.c-244 {fill: #808080}
|
||||
.c-245 {fill: #8a8a8a}
|
||||
.c-246 {fill: #949494}
|
||||
.c-247 {fill: #9e9e9e}
|
||||
.c-248 {fill: #a8a8a8}
|
||||
.c-249 {fill: #b2b2b2}
|
||||
.c-250 {fill: #bcbcbc}
|
||||
.c-251 {fill: #c6c6c6}
|
||||
.c-252 {fill: #d0d0d0}
|
||||
.c-253 {fill: #dadada}
|
||||
.c-254 {fill: #e4e4e4}
|
||||
.c-255 {fill: #eeeeee}
|
||||
|
||||
.br { font-weight: bold }
|
||||
.it { font-style: italic }
|
||||
.un { text-decoration: underline }
|
||||
</style>
|
||||
|
||||
<rect width="100%" height="100%" class="default-bg-fill" />
|
||||
|
||||
<svg x="0.625%" y="1.136%" class="default-text-fill">
|
||||
<g style="shape-rendering: optimizeSpeed">
|
||||
<rect x="5.625%" y="6.818%" width="0.625%" height="19.7" class="c-7" />
|
||||
<rect x="0.000%" y="95.455%" width="98.750%" height="19.7" class="c-2" />
|
||||
</g>
|
||||
|
||||
<text class="default-text-fill">
|
||||
<tspan y="0.000%">
|
||||
<tspan dy="1em" x="0.000%">c</tspan><tspan x="0.625%">h</tspan><tspan x="1.250%">a</tspan><tspan x="1.875%">o</tspan><tspan x="2.500%">s</tspan><tspan x="3.125%">:</tspan><tspan x="3.750%">~</tspan><tspan x="4.375%">$</tspan><tspan x="5.625%">#</tspan><tspan x="6.875%">B</tspan><tspan x="7.500%">u</tspan><tspan x="8.125%">i</tspan><tspan x="8.750%">l</tspan><tspan x="9.375%">d</tspan><tspan x="10.625%">a</tspan><tspan x="11.250%">n</tspan><tspan x="11.875%">d</tspan><tspan x="13.125%">a</tspan><tspan x="13.750%">p</tspan><tspan x="14.375%">p</tspan><tspan x="15.000%">l</tspan><tspan x="15.625%">y</tspan><tspan x="16.875%">C</tspan><tspan x="17.500%">h</tspan><tspan x="18.125%">a</tspan><tspan x="18.750%">o</tspan><tspan x="19.375%">s</tspan><tspan x="20.000%">E</tspan><tspan x="20.625%">n</tspan><tspan x="21.250%">g</tspan><tspan x="21.875%">i</tspan><tspan x="22.500%">n</tspan><tspan x="23.125%">e</tspan><tspan x="24.375%">C</tspan><tspan x="25.000%">R</tspan><tspan x="26.250%">t</tspan><tspan x="26.875%">o</tspan><tspan x="28.125%">u</tspan><tspan x="28.750%">n</tspan><tspan x="29.375%">l</tspan><tspan x="30.000%">e</tspan><tspan x="30.625%">a</tspan><tspan x="31.250%">s</tspan><tspan x="31.875%">h</tspan><tspan x="33.125%">C</tspan><tspan x="33.750%">h</tspan><tspan x="34.375%">a</tspan><tspan x="35.000%">o</tspan><tspan x="35.625%">s</tspan><tspan x="49.375%" class="c-2">│</tspan><tspan x="50.000%">E</tspan><tspan x="50.625%">v</tspan><tspan x="51.250%">e</tspan><tspan x="51.875%">r</tspan><tspan x="52.500%">y</tspan><tspan x="53.750%">1</tspan><tspan x="54.375%">.</tspan><tspan x="55.000%">0</tspan><tspan x="55.625%">s</tspan><tspan x="56.250%">:</tspan><tspan x="57.500%">k</tspan><tspan x="58.125%">u</tspan><tspan x="58.750%">b</tspan><tspan x="59.375%">e</tspan><tspan x="60.000%">c</tspan><tspan x="60.625%">t</tspan><tspan x="61.250%">l</tspan><tspan x="62.500%">g</tspan><tspan x="63.125%">e</tspan><tspan x="63.750%">t</tspan><tspan x="65.000%">p</tspan><tspan x="65.625%">o</tspan><tspan x="83.750%">F</tspan><tspan x="84.375%">r</tspan><tspan x="85.000%">i</tspan><tspan x="86.250%">O</tspan><tspan x="86.875%">c</tspan><tspan x="87.500%">t</tspan><tspan x="89.375%">4</tspan><tspan x="90.625%">1</tspan><tspan x="91.250%">9</tspan><tspan x="91.875%">:</tspan><tspan x="92.500%">3</tspan><tspan x="93.125%">2</tspan><tspan x="93.750%">:</tspan><tspan x="94.375%">3</tspan><tspan x="95.000%">5</tspan><tspan x="96.250%">2</tspan><tspan x="96.875%">0</tspan><tspan x="97.500%">1</tspan><tspan x="98.125%">9</tspan>
|
||||
</tspan>
|
||||
<tspan y="2.273%">
|
||||
<tspan dy="1em" x="0.000%">c</tspan><tspan x="0.625%">h</tspan><tspan x="1.250%">a</tspan><tspan x="1.875%">o</tspan><tspan x="2.500%">s</tspan><tspan x="3.125%">:</tspan><tspan x="3.750%">~</tspan><tspan x="4.375%">$</tspan><tspan x="49.375%" class="c-2">│</tspan>
|
||||
</tspan>
|
||||
<tspan y="4.545%">
|
||||
<tspan dy="1em" x="0.000%">c</tspan><tspan x="0.625%">h</tspan><tspan x="1.250%">a</tspan><tspan x="1.875%">o</tspan><tspan x="2.500%">s</tspan><tspan x="3.125%">:</tspan><tspan x="3.750%">~</tspan><tspan x="4.375%">$</tspan><tspan x="5.625%">v</tspan><tspan x="6.250%">i</tspan><tspan x="7.500%">c</tspan><tspan x="8.125%">h</tspan><tspan x="8.750%">a</tspan><tspan x="9.375%">o</tspan><tspan x="10.000%">s</tspan><tspan x="10.625%">e</tspan><tspan x="11.250%">n</tspan><tspan x="11.875%">g</tspan><tspan x="12.500%">i</tspan><tspan x="13.125%">n</tspan><tspan x="13.750%">e</tspan><tspan x="14.375%">.</tspan><tspan x="15.000%">y</tspan><tspan x="15.625%">a</tspan><tspan x="16.250%">m</tspan><tspan x="16.875%">l</tspan><tspan x="49.375%" class="c-2">│</tspan><tspan x="50.000%">N</tspan><tspan x="50.625%">A</tspan><tspan x="51.250%">M</tspan><tspan x="51.875%">E</tspan><tspan x="69.375%">R</tspan><tspan x="70.000%">E</tspan><tspan x="70.625%">A</tspan><tspan x="71.250%">D</tspan><tspan x="71.875%">Y</tspan><tspan x="74.375%">S</tspan><tspan x="75.000%">T</tspan><tspan x="75.625%">A</tspan><tspan x="76.250%">T</tspan><tspan x="76.875%">U</tspan><tspan x="77.500%">S</tspan><tspan x="80.625%">R</tspan><tspan x="81.250%">E</tspan><tspan x="81.875%">S</tspan><tspan x="82.500%">T</tspan><tspan x="83.125%">A</tspan><tspan x="83.750%">R</tspan><tspan x="84.375%">T</tspan><tspan x="85.000%">S</tspan><tspan x="87.500%">A</tspan><tspan x="88.125%">G</tspan><tspan x="88.750%">E</tspan>
|
||||
</tspan>
|
||||
<tspan y="6.818%">
|
||||
<tspan dy="1em" x="0.000%">c</tspan><tspan x="0.625%">h</tspan><tspan x="1.250%">a</tspan><tspan x="1.875%">o</tspan><tspan x="2.500%">s</tspan><tspan x="3.125%">:</tspan><tspan x="3.750%">~</tspan><tspan x="4.375%">$</tspan><tspan x="49.375%" class="c-2">│</tspan><tspan x="50.000%">h</tspan><tspan x="50.625%">e</tspan><tspan x="51.250%">l</tspan><tspan x="51.875%">l</tspan><tspan x="52.500%">o</tspan><tspan x="53.125%">-</tspan><tspan x="53.750%">d</tspan><tspan x="54.375%">e</tspan><tspan x="55.000%">p</tspan><tspan x="55.625%">l</tspan><tspan x="56.250%">o</tspan><tspan x="56.875%">y</tspan><tspan x="57.500%">-</tspan><tspan x="58.125%">d</tspan><tspan x="58.750%">d</tspan><tspan x="59.375%">5</tspan><tspan x="60.000%">9</tspan><tspan x="60.625%">b</tspan><tspan x="61.250%">8</tspan><tspan x="61.875%">9</tspan><tspan x="62.500%">5</tspan><tspan x="63.125%">6</tspan><tspan x="63.750%">-</tspan><tspan x="64.375%">h</tspan><tspan x="65.000%">x</tspan><tspan x="65.625%">c</tspan><tspan x="66.250%">j</tspan><tspan x="66.875%">v</tspan><tspan x="69.375%">1</tspan><tspan x="70.000%">/</tspan><tspan x="70.625%">1</tspan><tspan x="74.375%">R</tspan><tspan x="75.000%">u</tspan><tspan x="75.625%">n</tspan><tspan x="76.250%">n</tspan><tspan x="76.875%">i</tspan><tspan x="77.500%">n</tspan><tspan x="78.125%">g</tspan><tspan x="80.625%">0</tspan><tspan x="87.500%">1</tspan><tspan x="88.125%">9</tspan><tspan x="88.750%">m</tspan>
|
||||
</tspan>
|
||||
<tspan y="9.091%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan>
|
||||
</tspan>
|
||||
<tspan y="11.364%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan>
|
||||
</tspan>
|
||||
<tspan y="13.636%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan>
|
||||
</tspan>
|
||||
<tspan y="15.909%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan>
|
||||
</tspan>
|
||||
<tspan y="18.182%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan>
|
||||
</tspan>
|
||||
<tspan y="20.455%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan>
|
||||
</tspan>
|
||||
<tspan y="22.727%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan>
|
||||
</tspan>
|
||||
<tspan y="25.000%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan>
|
||||
</tspan>
|
||||
<tspan y="27.273%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan>
|
||||
</tspan>
|
||||
<tspan y="29.545%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan>
|
||||
</tspan>
|
||||
<tspan y="31.818%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan>
|
||||
</tspan>
|
||||
<tspan y="34.091%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan>
|
||||
</tspan>
|
||||
<tspan y="36.364%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan>
|
||||
</tspan>
|
||||
<tspan y="38.636%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan>
|
||||
</tspan>
|
||||
<tspan y="40.909%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan>
|
||||
</tspan>
|
||||
<tspan y="43.182%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan>
|
||||
</tspan>
|
||||
<tspan y="45.455%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan>
|
||||
</tspan>
|
||||
<tspan y="47.727%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">├</tspan><tspan x="50.000%">─</tspan><tspan x="50.625%">─</tspan><tspan x="51.250%">─</tspan><tspan x="51.875%">─</tspan><tspan x="52.500%">─</tspan><tspan x="53.125%">─</tspan><tspan x="53.750%">─</tspan><tspan x="54.375%">─</tspan><tspan x="55.000%">─</tspan><tspan x="55.625%">─</tspan><tspan x="56.250%">─</tspan><tspan x="56.875%">─</tspan><tspan x="57.500%">─</tspan><tspan x="58.125%">─</tspan><tspan x="58.750%">─</tspan><tspan x="59.375%">─</tspan><tspan x="60.000%">─</tspan><tspan x="60.625%">─</tspan><tspan x="61.250%">─</tspan><tspan x="61.875%">─</tspan><tspan x="62.500%">─</tspan><tspan x="63.125%">─</tspan><tspan x="63.750%">─</tspan><tspan x="64.375%">─</tspan><tspan x="65.000%">─</tspan><tspan x="65.625%">─</tspan><tspan x="66.250%">─</tspan><tspan x="66.875%">─</tspan><tspan x="67.500%">─</tspan><tspan x="68.125%">─</tspan><tspan x="68.750%">─</tspan><tspan x="69.375%">─</tspan><tspan x="70.000%">─</tspan><tspan x="70.625%">─</tspan><tspan x="71.250%">─</tspan><tspan x="71.875%">─</tspan><tspan x="72.500%">─</tspan><tspan x="73.125%">─</tspan><tspan x="73.750%">─</tspan><tspan x="74.375%">─</tspan><tspan x="75.000%">─</tspan><tspan x="75.625%">─</tspan><tspan x="76.250%">─</tspan><tspan x="76.875%">─</tspan><tspan x="77.500%">─</tspan><tspan x="78.125%">─</tspan><tspan x="78.750%">─</tspan><tspan x="79.375%">─</tspan><tspan x="80.000%">─</tspan><tspan x="80.625%">─</tspan><tspan x="81.250%">─</tspan><tspan x="81.875%">─</tspan><tspan x="82.500%">─</tspan><tspan x="83.125%">─</tspan><tspan x="83.750%">─</tspan><tspan x="84.375%">─</tspan><tspan x="85.000%">─</tspan><tspan x="85.625%">─</tspan><tspan x="86.250%">─</tspan><tspan x="86.875%">─</tspan><tspan x="87.500%">─</tspan><tspan x="88.125%">─</tspan><tspan x="88.750%">─</tspan><tspan x="89.375%">─</tspan><tspan x="90.000%">─</tspan><tspan x="90.625%">─</tspan><tspan x="91.250%">─</tspan><tspan x="91.875%">─</tspan><tspan x="92.500%">─</tspan><tspan x="93.125%">─</tspan><tspan x="93.750%">─</tspan><tspan x="94.375%">─</tspan><tspan x="95.000%">─</tspan><tspan x="95.625%">─</tspan><tspan x="96.250%">─</tspan><tspan x="96.875%">─</tspan><tspan x="97.500%">─</tspan><tspan x="98.125%">─</tspan>
|
||||
</tspan>
|
||||
<tspan y="50.000%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan><tspan x="50.000%" class="br c-10">H</tspan><tspan x="50.625%" class="br c-10">e</tspan><tspan x="51.250%" class="br c-10">l</tspan><tspan x="51.875%" class="br c-10">l</tspan><tspan x="52.500%" class="br c-10">o</tspan><tspan x="53.750%" class="br c-10">W</tspan><tspan x="54.375%" class="br c-10">o</tspan><tspan x="55.000%" class="br c-10">r</tspan><tspan x="55.625%" class="br c-10">l</tspan><tspan x="56.250%" class="br c-10">d</tspan><tspan x="57.500%" class="br c-10">i</tspan><tspan x="58.125%" class="br c-10">s</tspan><tspan x="59.375%" class="br c-10">o</tspan><tspan x="60.000%" class="br c-10">n</tspan><tspan x="60.625%" class="br c-10">l</tspan><tspan x="61.250%" class="br c-10">i</tspan><tspan x="61.875%" class="br c-10">n</tspan><tspan x="62.500%" class="br c-10">e</tspan><tspan x="63.750%">H</tspan><tspan x="64.375%">T</tspan><tspan x="65.000%">T</tspan><tspan x="65.625%">P</tspan><tspan x="66.250%">/</tspan><tspan x="66.875%">2</tspan><tspan x="68.125%">2</tspan><tspan x="68.750%">0</tspan><tspan x="69.375%">0</tspan><tspan x="70.625%">O</tspan><tspan x="71.250%">K</tspan>
|
||||
</tspan>
|
||||
<tspan y="52.273%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan><tspan x="50.000%" class="br c-10">H</tspan><tspan x="50.625%" class="br c-10">e</tspan><tspan x="51.250%" class="br c-10">l</tspan><tspan x="51.875%" class="br c-10">l</tspan><tspan x="52.500%" class="br c-10">o</tspan><tspan x="53.750%" class="br c-10">W</tspan><tspan x="54.375%" class="br c-10">o</tspan><tspan x="55.000%" class="br c-10">r</tspan><tspan x="55.625%" class="br c-10">l</tspan><tspan x="56.250%" class="br c-10">d</tspan><tspan x="57.500%" class="br c-10">i</tspan><tspan x="58.125%" class="br c-10">s</tspan><tspan x="59.375%" class="br c-10">o</tspan><tspan x="60.000%" class="br c-10">n</tspan><tspan x="60.625%" class="br c-10">l</tspan><tspan x="61.250%" class="br c-10">i</tspan><tspan x="61.875%" class="br c-10">n</tspan><tspan x="62.500%" class="br c-10">e</tspan><tspan x="63.750%">H</tspan><tspan x="64.375%">T</tspan><tspan x="65.000%">T</tspan><tspan x="65.625%">P</tspan><tspan x="66.250%">/</tspan><tspan x="66.875%">2</tspan><tspan x="68.125%">2</tspan><tspan x="68.750%">0</tspan><tspan x="69.375%">0</tspan><tspan x="70.625%">O</tspan><tspan x="71.250%">K</tspan>
|
||||
</tspan>
|
||||
<tspan y="54.545%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan><tspan x="50.000%" class="br c-10">H</tspan><tspan x="50.625%" class="br c-10">e</tspan><tspan x="51.250%" class="br c-10">l</tspan><tspan x="51.875%" class="br c-10">l</tspan><tspan x="52.500%" class="br c-10">o</tspan><tspan x="53.750%" class="br c-10">W</tspan><tspan x="54.375%" class="br c-10">o</tspan><tspan x="55.000%" class="br c-10">r</tspan><tspan x="55.625%" class="br c-10">l</tspan><tspan x="56.250%" class="br c-10">d</tspan><tspan x="57.500%" class="br c-10">i</tspan><tspan x="58.125%" class="br c-10">s</tspan><tspan x="59.375%" class="br c-10">o</tspan><tspan x="60.000%" class="br c-10">n</tspan><tspan x="60.625%" class="br c-10">l</tspan><tspan x="61.250%" class="br c-10">i</tspan><tspan x="61.875%" class="br c-10">n</tspan><tspan x="62.500%" class="br c-10">e</tspan><tspan x="63.750%">H</tspan><tspan x="64.375%">T</tspan><tspan x="65.000%">T</tspan><tspan x="65.625%">P</tspan><tspan x="66.250%">/</tspan><tspan x="66.875%">2</tspan><tspan x="68.125%">2</tspan><tspan x="68.750%">0</tspan><tspan x="69.375%">0</tspan><tspan x="70.625%">O</tspan><tspan x="71.250%">K</tspan>
|
||||
</tspan>
|
||||
<tspan y="56.818%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan><tspan x="50.000%" class="br c-10">H</tspan><tspan x="50.625%" class="br c-10">e</tspan><tspan x="51.250%" class="br c-10">l</tspan><tspan x="51.875%" class="br c-10">l</tspan><tspan x="52.500%" class="br c-10">o</tspan><tspan x="53.750%" class="br c-10">W</tspan><tspan x="54.375%" class="br c-10">o</tspan><tspan x="55.000%" class="br c-10">r</tspan><tspan x="55.625%" class="br c-10">l</tspan><tspan x="56.250%" class="br c-10">d</tspan><tspan x="57.500%" class="br c-10">i</tspan><tspan x="58.125%" class="br c-10">s</tspan><tspan x="59.375%" class="br c-10">o</tspan><tspan x="60.000%" class="br c-10">n</tspan><tspan x="60.625%" class="br c-10">l</tspan><tspan x="61.250%" class="br c-10">i</tspan><tspan x="61.875%" class="br c-10">n</tspan><tspan x="62.500%" class="br c-10">e</tspan><tspan x="63.750%">H</tspan><tspan x="64.375%">T</tspan><tspan x="65.000%">T</tspan><tspan x="65.625%">P</tspan><tspan x="66.250%">/</tspan><tspan x="66.875%">2</tspan><tspan x="68.125%">2</tspan><tspan x="68.750%">0</tspan><tspan x="69.375%">0</tspan><tspan x="70.625%">O</tspan><tspan x="71.250%">K</tspan>
|
||||
</tspan>
|
||||
<tspan y="59.091%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan><tspan x="50.000%" class="br c-10">H</tspan><tspan x="50.625%" class="br c-10">e</tspan><tspan x="51.250%" class="br c-10">l</tspan><tspan x="51.875%" class="br c-10">l</tspan><tspan x="52.500%" class="br c-10">o</tspan><tspan x="53.750%" class="br c-10">W</tspan><tspan x="54.375%" class="br c-10">o</tspan><tspan x="55.000%" class="br c-10">r</tspan><tspan x="55.625%" class="br c-10">l</tspan><tspan x="56.250%" class="br c-10">d</tspan><tspan x="57.500%" class="br c-10">i</tspan><tspan x="58.125%" class="br c-10">s</tspan><tspan x="59.375%" class="br c-10">o</tspan><tspan x="60.000%" class="br c-10">n</tspan><tspan x="60.625%" class="br c-10">l</tspan><tspan x="61.250%" class="br c-10">i</tspan><tspan x="61.875%" class="br c-10">n</tspan><tspan x="62.500%" class="br c-10">e</tspan><tspan x="63.750%">H</tspan><tspan x="64.375%">T</tspan><tspan x="65.000%">T</tspan><tspan x="65.625%">P</tspan><tspan x="66.250%">/</tspan><tspan x="66.875%">2</tspan><tspan x="68.125%">2</tspan><tspan x="68.750%">0</tspan><tspan x="69.375%">0</tspan><tspan x="70.625%">O</tspan><tspan x="71.250%">K</tspan>
|
||||
</tspan>
|
||||
<tspan y="61.364%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan><tspan x="50.000%" class="br c-10">H</tspan><tspan x="50.625%" class="br c-10">e</tspan><tspan x="51.250%" class="br c-10">l</tspan><tspan x="51.875%" class="br c-10">l</tspan><tspan x="52.500%" class="br c-10">o</tspan><tspan x="53.750%" class="br c-10">W</tspan><tspan x="54.375%" class="br c-10">o</tspan><tspan x="55.000%" class="br c-10">r</tspan><tspan x="55.625%" class="br c-10">l</tspan><tspan x="56.250%" class="br c-10">d</tspan><tspan x="57.500%" class="br c-10">i</tspan><tspan x="58.125%" class="br c-10">s</tspan><tspan x="59.375%" class="br c-10">o</tspan><tspan x="60.000%" class="br c-10">n</tspan><tspan x="60.625%" class="br c-10">l</tspan><tspan x="61.250%" class="br c-10">i</tspan><tspan x="61.875%" class="br c-10">n</tspan><tspan x="62.500%" class="br c-10">e</tspan><tspan x="63.750%">H</tspan><tspan x="64.375%">T</tspan><tspan x="65.000%">T</tspan><tspan x="65.625%">P</tspan><tspan x="66.250%">/</tspan><tspan x="66.875%">2</tspan><tspan x="68.125%">2</tspan><tspan x="68.750%">0</tspan><tspan x="69.375%">0</tspan><tspan x="70.625%">O</tspan><tspan x="71.250%">K</tspan>
|
||||
</tspan>
|
||||
<tspan y="63.636%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan><tspan x="50.000%" class="br c-10">H</tspan><tspan x="50.625%" class="br c-10">e</tspan><tspan x="51.250%" class="br c-10">l</tspan><tspan x="51.875%" class="br c-10">l</tspan><tspan x="52.500%" class="br c-10">o</tspan><tspan x="53.750%" class="br c-10">W</tspan><tspan x="54.375%" class="br c-10">o</tspan><tspan x="55.000%" class="br c-10">r</tspan><tspan x="55.625%" class="br c-10">l</tspan><tspan x="56.250%" class="br c-10">d</tspan><tspan x="57.500%" class="br c-10">i</tspan><tspan x="58.125%" class="br c-10">s</tspan><tspan x="59.375%" class="br c-10">o</tspan><tspan x="60.000%" class="br c-10">n</tspan><tspan x="60.625%" class="br c-10">l</tspan><tspan x="61.250%" class="br c-10">i</tspan><tspan x="61.875%" class="br c-10">n</tspan><tspan x="62.500%" class="br c-10">e</tspan><tspan x="63.750%">H</tspan><tspan x="64.375%">T</tspan><tspan x="65.000%">T</tspan><tspan x="65.625%">P</tspan><tspan x="66.250%">/</tspan><tspan x="66.875%">2</tspan><tspan x="68.125%">2</tspan><tspan x="68.750%">0</tspan><tspan x="69.375%">0</tspan><tspan x="70.625%">O</tspan><tspan x="71.250%">K</tspan>
|
||||
</tspan>
|
||||
<tspan y="65.909%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan><tspan x="50.000%" class="br c-10">H</tspan><tspan x="50.625%" class="br c-10">e</tspan><tspan x="51.250%" class="br c-10">l</tspan><tspan x="51.875%" class="br c-10">l</tspan><tspan x="52.500%" class="br c-10">o</tspan><tspan x="53.750%" class="br c-10">W</tspan><tspan x="54.375%" class="br c-10">o</tspan><tspan x="55.000%" class="br c-10">r</tspan><tspan x="55.625%" class="br c-10">l</tspan><tspan x="56.250%" class="br c-10">d</tspan><tspan x="57.500%" class="br c-10">i</tspan><tspan x="58.125%" class="br c-10">s</tspan><tspan x="59.375%" class="br c-10">o</tspan><tspan x="60.000%" class="br c-10">n</tspan><tspan x="60.625%" class="br c-10">l</tspan><tspan x="61.250%" class="br c-10">i</tspan><tspan x="61.875%" class="br c-10">n</tspan><tspan x="62.500%" class="br c-10">e</tspan><tspan x="63.750%">H</tspan><tspan x="64.375%">T</tspan><tspan x="65.000%">T</tspan><tspan x="65.625%">P</tspan><tspan x="66.250%">/</tspan><tspan x="66.875%">2</tspan><tspan x="68.125%">2</tspan><tspan x="68.750%">0</tspan><tspan x="69.375%">0</tspan><tspan x="70.625%">O</tspan><tspan x="71.250%">K</tspan>
|
||||
</tspan>
|
||||
<tspan y="68.182%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan><tspan x="50.000%" class="br c-10">H</tspan><tspan x="50.625%" class="br c-10">e</tspan><tspan x="51.250%" class="br c-10">l</tspan><tspan x="51.875%" class="br c-10">l</tspan><tspan x="52.500%" class="br c-10">o</tspan><tspan x="53.750%" class="br c-10">W</tspan><tspan x="54.375%" class="br c-10">o</tspan><tspan x="55.000%" class="br c-10">r</tspan><tspan x="55.625%" class="br c-10">l</tspan><tspan x="56.250%" class="br c-10">d</tspan><tspan x="57.500%" class="br c-10">i</tspan><tspan x="58.125%" class="br c-10">s</tspan><tspan x="59.375%" class="br c-10">o</tspan><tspan x="60.000%" class="br c-10">n</tspan><tspan x="60.625%" class="br c-10">l</tspan><tspan x="61.250%" class="br c-10">i</tspan><tspan x="61.875%" class="br c-10">n</tspan><tspan x="62.500%" class="br c-10">e</tspan><tspan x="63.750%">H</tspan><tspan x="64.375%">T</tspan><tspan x="65.000%">T</tspan><tspan x="65.625%">P</tspan><tspan x="66.250%">/</tspan><tspan x="66.875%">2</tspan><tspan x="68.125%">2</tspan><tspan x="68.750%">0</tspan><tspan x="69.375%">0</tspan><tspan x="70.625%">O</tspan><tspan x="71.250%">K</tspan>
|
||||
</tspan>
|
||||
<tspan y="70.455%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan><tspan x="50.000%" class="br c-10">H</tspan><tspan x="50.625%" class="br c-10">e</tspan><tspan x="51.250%" class="br c-10">l</tspan><tspan x="51.875%" class="br c-10">l</tspan><tspan x="52.500%" class="br c-10">o</tspan><tspan x="53.750%" class="br c-10">W</tspan><tspan x="54.375%" class="br c-10">o</tspan><tspan x="55.000%" class="br c-10">r</tspan><tspan x="55.625%" class="br c-10">l</tspan><tspan x="56.250%" class="br c-10">d</tspan><tspan x="57.500%" class="br c-10">i</tspan><tspan x="58.125%" class="br c-10">s</tspan><tspan x="59.375%" class="br c-10">o</tspan><tspan x="60.000%" class="br c-10">n</tspan><tspan x="60.625%" class="br c-10">l</tspan><tspan x="61.250%" class="br c-10">i</tspan><tspan x="61.875%" class="br c-10">n</tspan><tspan x="62.500%" class="br c-10">e</tspan><tspan x="63.750%">H</tspan><tspan x="64.375%">T</tspan><tspan x="65.000%">T</tspan><tspan x="65.625%">P</tspan><tspan x="66.250%">/</tspan><tspan x="66.875%">2</tspan><tspan x="68.125%">2</tspan><tspan x="68.750%">0</tspan><tspan x="69.375%">0</tspan><tspan x="70.625%">O</tspan><tspan x="71.250%">K</tspan>
|
||||
</tspan>
|
||||
<tspan y="72.727%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan><tspan x="50.000%" class="br c-10">H</tspan><tspan x="50.625%" class="br c-10">e</tspan><tspan x="51.250%" class="br c-10">l</tspan><tspan x="51.875%" class="br c-10">l</tspan><tspan x="52.500%" class="br c-10">o</tspan><tspan x="53.750%" class="br c-10">W</tspan><tspan x="54.375%" class="br c-10">o</tspan><tspan x="55.000%" class="br c-10">r</tspan><tspan x="55.625%" class="br c-10">l</tspan><tspan x="56.250%" class="br c-10">d</tspan><tspan x="57.500%" class="br c-10">i</tspan><tspan x="58.125%" class="br c-10">s</tspan><tspan x="59.375%" class="br c-10">o</tspan><tspan x="60.000%" class="br c-10">n</tspan><tspan x="60.625%" class="br c-10">l</tspan><tspan x="61.250%" class="br c-10">i</tspan><tspan x="61.875%" class="br c-10">n</tspan><tspan x="62.500%" class="br c-10">e</tspan><tspan x="63.750%">H</tspan><tspan x="64.375%">T</tspan><tspan x="65.000%">T</tspan><tspan x="65.625%">P</tspan><tspan x="66.250%">/</tspan><tspan x="66.875%">2</tspan><tspan x="68.125%">2</tspan><tspan x="68.750%">0</tspan><tspan x="69.375%">0</tspan><tspan x="70.625%">O</tspan><tspan x="71.250%">K</tspan>
|
||||
</tspan>
|
||||
<tspan y="75.000%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan><tspan x="50.000%" class="br c-10">H</tspan><tspan x="50.625%" class="br c-10">e</tspan><tspan x="51.250%" class="br c-10">l</tspan><tspan x="51.875%" class="br c-10">l</tspan><tspan x="52.500%" class="br c-10">o</tspan><tspan x="53.750%" class="br c-10">W</tspan><tspan x="54.375%" class="br c-10">o</tspan><tspan x="55.000%" class="br c-10">r</tspan><tspan x="55.625%" class="br c-10">l</tspan><tspan x="56.250%" class="br c-10">d</tspan><tspan x="57.500%" class="br c-10">i</tspan><tspan x="58.125%" class="br c-10">s</tspan><tspan x="59.375%" class="br c-10">o</tspan><tspan x="60.000%" class="br c-10">n</tspan><tspan x="60.625%" class="br c-10">l</tspan><tspan x="61.250%" class="br c-10">i</tspan><tspan x="61.875%" class="br c-10">n</tspan><tspan x="62.500%" class="br c-10">e</tspan><tspan x="63.750%">H</tspan><tspan x="64.375%">T</tspan><tspan x="65.000%">T</tspan><tspan x="65.625%">P</tspan><tspan x="66.250%">/</tspan><tspan x="66.875%">2</tspan><tspan x="68.125%">2</tspan><tspan x="68.750%">0</tspan><tspan x="69.375%">0</tspan><tspan x="70.625%">O</tspan><tspan x="71.250%">K</tspan>
|
||||
</tspan>
|
||||
<tspan y="77.273%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan><tspan x="50.000%" class="br c-10">H</tspan><tspan x="50.625%" class="br c-10">e</tspan><tspan x="51.250%" class="br c-10">l</tspan><tspan x="51.875%" class="br c-10">l</tspan><tspan x="52.500%" class="br c-10">o</tspan><tspan x="53.750%" class="br c-10">W</tspan><tspan x="54.375%" class="br c-10">o</tspan><tspan x="55.000%" class="br c-10">r</tspan><tspan x="55.625%" class="br c-10">l</tspan><tspan x="56.250%" class="br c-10">d</tspan><tspan x="57.500%" class="br c-10">i</tspan><tspan x="58.125%" class="br c-10">s</tspan><tspan x="59.375%" class="br c-10">o</tspan><tspan x="60.000%" class="br c-10">n</tspan><tspan x="60.625%" class="br c-10">l</tspan><tspan x="61.250%" class="br c-10">i</tspan><tspan x="61.875%" class="br c-10">n</tspan><tspan x="62.500%" class="br c-10">e</tspan><tspan x="63.750%">H</tspan><tspan x="64.375%">T</tspan><tspan x="65.000%">T</tspan><tspan x="65.625%">P</tspan><tspan x="66.250%">/</tspan><tspan x="66.875%">2</tspan><tspan x="68.125%">2</tspan><tspan x="68.750%">0</tspan><tspan x="69.375%">0</tspan><tspan x="70.625%">O</tspan><tspan x="71.250%">K</tspan>
|
||||
</tspan>
|
||||
<tspan y="79.545%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan><tspan x="50.000%" class="br c-10">H</tspan><tspan x="50.625%" class="br c-10">e</tspan><tspan x="51.250%" class="br c-10">l</tspan><tspan x="51.875%" class="br c-10">l</tspan><tspan x="52.500%" class="br c-10">o</tspan><tspan x="53.750%" class="br c-10">W</tspan><tspan x="54.375%" class="br c-10">o</tspan><tspan x="55.000%" class="br c-10">r</tspan><tspan x="55.625%" class="br c-10">l</tspan><tspan x="56.250%" class="br c-10">d</tspan><tspan x="57.500%" class="br c-10">i</tspan><tspan x="58.125%" class="br c-10">s</tspan><tspan x="59.375%" class="br c-10">o</tspan><tspan x="60.000%" class="br c-10">n</tspan><tspan x="60.625%" class="br c-10">l</tspan><tspan x="61.250%" class="br c-10">i</tspan><tspan x="61.875%" class="br c-10">n</tspan><tspan x="62.500%" class="br c-10">e</tspan><tspan x="63.750%">H</tspan><tspan x="64.375%">T</tspan><tspan x="65.000%">T</tspan><tspan x="65.625%">P</tspan><tspan x="66.250%">/</tspan><tspan x="66.875%">2</tspan><tspan x="68.125%">2</tspan><tspan x="68.750%">0</tspan><tspan x="69.375%">0</tspan><tspan x="70.625%">O</tspan><tspan x="71.250%">K</tspan>
|
||||
</tspan>
|
||||
<tspan y="81.818%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan><tspan x="50.000%" class="br c-10">H</tspan><tspan x="50.625%" class="br c-10">e</tspan><tspan x="51.250%" class="br c-10">l</tspan><tspan x="51.875%" class="br c-10">l</tspan><tspan x="52.500%" class="br c-10">o</tspan><tspan x="53.750%" class="br c-10">W</tspan><tspan x="54.375%" class="br c-10">o</tspan><tspan x="55.000%" class="br c-10">r</tspan><tspan x="55.625%" class="br c-10">l</tspan><tspan x="56.250%" class="br c-10">d</tspan><tspan x="57.500%" class="br c-10">i</tspan><tspan x="58.125%" class="br c-10">s</tspan><tspan x="59.375%" class="br c-10">o</tspan><tspan x="60.000%" class="br c-10">n</tspan><tspan x="60.625%" class="br c-10">l</tspan><tspan x="61.250%" class="br c-10">i</tspan><tspan x="61.875%" class="br c-10">n</tspan><tspan x="62.500%" class="br c-10">e</tspan><tspan x="63.750%">H</tspan><tspan x="64.375%">T</tspan><tspan x="65.000%">T</tspan><tspan x="65.625%">P</tspan><tspan x="66.250%">/</tspan><tspan x="66.875%">2</tspan><tspan x="68.125%">2</tspan><tspan x="68.750%">0</tspan><tspan x="69.375%">0</tspan><tspan x="70.625%">O</tspan><tspan x="71.250%">K</tspan>
|
||||
</tspan>
|
||||
<tspan y="84.091%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan><tspan x="50.000%" class="br c-10">H</tspan><tspan x="50.625%" class="br c-10">e</tspan><tspan x="51.250%" class="br c-10">l</tspan><tspan x="51.875%" class="br c-10">l</tspan><tspan x="52.500%" class="br c-10">o</tspan><tspan x="53.750%" class="br c-10">W</tspan><tspan x="54.375%" class="br c-10">o</tspan><tspan x="55.000%" class="br c-10">r</tspan><tspan x="55.625%" class="br c-10">l</tspan><tspan x="56.250%" class="br c-10">d</tspan><tspan x="57.500%" class="br c-10">i</tspan><tspan x="58.125%" class="br c-10">s</tspan><tspan x="59.375%" class="br c-10">o</tspan><tspan x="60.000%" class="br c-10">n</tspan><tspan x="60.625%" class="br c-10">l</tspan><tspan x="61.250%" class="br c-10">i</tspan><tspan x="61.875%" class="br c-10">n</tspan><tspan x="62.500%" class="br c-10">e</tspan><tspan x="63.750%">H</tspan><tspan x="64.375%">T</tspan><tspan x="65.000%">T</tspan><tspan x="65.625%">P</tspan><tspan x="66.250%">/</tspan><tspan x="66.875%">2</tspan><tspan x="68.125%">2</tspan><tspan x="68.750%">0</tspan><tspan x="69.375%">0</tspan><tspan x="70.625%">O</tspan><tspan x="71.250%">K</tspan>
|
||||
</tspan>
|
||||
<tspan y="86.364%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan><tspan x="50.000%" class="br c-10">H</tspan><tspan x="50.625%" class="br c-10">e</tspan><tspan x="51.250%" class="br c-10">l</tspan><tspan x="51.875%" class="br c-10">l</tspan><tspan x="52.500%" class="br c-10">o</tspan><tspan x="53.750%" class="br c-10">W</tspan><tspan x="54.375%" class="br c-10">o</tspan><tspan x="55.000%" class="br c-10">r</tspan><tspan x="55.625%" class="br c-10">l</tspan><tspan x="56.250%" class="br c-10">d</tspan><tspan x="57.500%" class="br c-10">i</tspan><tspan x="58.125%" class="br c-10">s</tspan><tspan x="59.375%" class="br c-10">o</tspan><tspan x="60.000%" class="br c-10">n</tspan><tspan x="60.625%" class="br c-10">l</tspan><tspan x="61.250%" class="br c-10">i</tspan><tspan x="61.875%" class="br c-10">n</tspan><tspan x="62.500%" class="br c-10">e</tspan><tspan x="63.750%">H</tspan><tspan x="64.375%">T</tspan><tspan x="65.000%">T</tspan><tspan x="65.625%">P</tspan><tspan x="66.250%">/</tspan><tspan x="66.875%">2</tspan><tspan x="68.125%">2</tspan><tspan x="68.750%">0</tspan><tspan x="69.375%">0</tspan><tspan x="70.625%">O</tspan><tspan x="71.250%">K</tspan>
|
||||
</tspan>
|
||||
<tspan y="88.636%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan><tspan x="50.000%" class="br c-10">H</tspan><tspan x="50.625%" class="br c-10">e</tspan><tspan x="51.250%" class="br c-10">l</tspan><tspan x="51.875%" class="br c-10">l</tspan><tspan x="52.500%" class="br c-10">o</tspan><tspan x="53.750%" class="br c-10">W</tspan><tspan x="54.375%" class="br c-10">o</tspan><tspan x="55.000%" class="br c-10">r</tspan><tspan x="55.625%" class="br c-10">l</tspan><tspan x="56.250%" class="br c-10">d</tspan><tspan x="57.500%" class="br c-10">i</tspan><tspan x="58.125%" class="br c-10">s</tspan><tspan x="59.375%" class="br c-10">o</tspan><tspan x="60.000%" class="br c-10">n</tspan><tspan x="60.625%" class="br c-10">l</tspan><tspan x="61.250%" class="br c-10">i</tspan><tspan x="61.875%" class="br c-10">n</tspan><tspan x="62.500%" class="br c-10">e</tspan><tspan x="63.750%">H</tspan><tspan x="64.375%">T</tspan><tspan x="65.000%">T</tspan><tspan x="65.625%">P</tspan><tspan x="66.250%">/</tspan><tspan x="66.875%">2</tspan><tspan x="68.125%">2</tspan><tspan x="68.750%">0</tspan><tspan x="69.375%">0</tspan><tspan x="70.625%">O</tspan><tspan x="71.250%">K</tspan>
|
||||
</tspan>
|
||||
<tspan y="90.909%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan><tspan x="50.000%" class="br c-10">H</tspan><tspan x="50.625%" class="br c-10">e</tspan><tspan x="51.250%" class="br c-10">l</tspan><tspan x="51.875%" class="br c-10">l</tspan><tspan x="52.500%" class="br c-10">o</tspan><tspan x="53.750%" class="br c-10">W</tspan><tspan x="54.375%" class="br c-10">o</tspan><tspan x="55.000%" class="br c-10">r</tspan><tspan x="55.625%" class="br c-10">l</tspan><tspan x="56.250%" class="br c-10">d</tspan><tspan x="57.500%" class="br c-10">i</tspan><tspan x="58.125%" class="br c-10">s</tspan><tspan x="59.375%" class="br c-10">o</tspan><tspan x="60.000%" class="br c-10">n</tspan><tspan x="60.625%" class="br c-10">l</tspan><tspan x="61.250%" class="br c-10">i</tspan><tspan x="61.875%" class="br c-10">n</tspan><tspan x="62.500%" class="br c-10">e</tspan><tspan x="63.750%">H</tspan><tspan x="64.375%">T</tspan><tspan x="65.000%">T</tspan><tspan x="65.625%">P</tspan><tspan x="66.250%">/</tspan><tspan x="66.875%">2</tspan><tspan x="68.125%">2</tspan><tspan x="68.750%">0</tspan><tspan x="69.375%">0</tspan><tspan x="70.625%">O</tspan><tspan x="71.250%">K</tspan>
|
||||
</tspan>
|
||||
<tspan y="93.182%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan>
|
||||
</tspan>
|
||||
<tspan y="95.455%">
|
||||
<tspan dy="1em" x="0.000%" class="c-0">[</tspan><tspan x="0.625%" class="c-0">d</tspan><tspan x="1.250%" class="c-0">e</tspan><tspan x="1.875%" class="c-0">m</tspan><tspan x="2.500%" class="c-0">o</tspan><tspan x="3.125%" class="c-0">]</tspan><tspan x="4.375%" class="c-0">0</tspan><tspan x="5.000%" class="c-0">:</tspan><tspan x="5.625%" class="c-0">s</tspan><tspan x="6.250%" class="c-0">s</tspan><tspan x="6.875%" class="c-0">h</tspan><tspan x="7.500%" class="c-0">*</tspan><tspan x="75.625%" class="c-0">"</tspan><tspan x="76.250%" class="c-0">r</tspan><tspan x="76.875%" class="c-0">a</tspan><tspan x="77.500%" class="c-0">h</tspan><tspan x="78.125%" class="c-0">u</tspan><tspan x="78.750%" class="c-0">l</tspan><tspan x="79.375%" class="c-0">-</tspan><tspan x="80.000%" class="c-0">T</tspan><tspan x="80.625%" class="c-0">h</tspan><tspan x="81.250%" class="c-0">i</tspan><tspan x="81.875%" class="c-0">n</tspan><tspan x="82.500%" class="c-0">k</tspan><tspan x="83.125%" class="c-0">P</tspan><tspan x="83.750%" class="c-0">a</tspan><tspan x="84.375%" class="c-0">d</tspan><tspan x="85.000%" class="c-0">-</tspan><tspan x="85.625%" class="c-0">E</tspan><tspan x="86.250%" class="c-0">4</tspan><tspan x="86.875%" class="c-0">9</tspan><tspan x="87.500%" class="c-0">0</tspan><tspan x="88.125%" class="c-0">"</tspan><tspan x="89.375%" class="c-0">0</tspan><tspan x="90.000%" class="c-0">1</tspan><tspan x="90.625%" class="c-0">:</tspan><tspan x="91.250%" class="c-0">0</tspan><tspan x="91.875%" class="c-0">2</tspan><tspan x="93.125%" class="c-0">0</tspan><tspan x="93.750%" class="c-0">5</tspan><tspan x="94.375%" class="c-0">-</tspan><tspan x="95.000%" class="c-0">O</tspan><tspan x="95.625%" class="c-0">c</tspan><tspan x="96.250%" class="c-0">t</tspan><tspan x="96.875%" class="c-0">-</tspan><tspan x="97.500%" class="c-0">1</tspan><tspan x="98.125%" class="c-0">9</tspan>
|
||||
</tspan>
|
||||
</text>
|
||||
|
||||
<g transform="translate(-50 -50)">
|
||||
<svg x="50%" y="50%" width="100" height="100">
|
||||
<svg version="1.1" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 866.0254037844387 866.0254037844387">
|
||||
<defs>
|
||||
<mask id="small-triangle-mask">
|
||||
<rect width="100%" height="100%" fill="white"/>
|
||||
<polygon points="508.01270189221935 433.01270189221935, 208.0127018922194 259.8076211353316, 208.01270189221927 606.217782649107" fill="black"></polygon>
|
||||
</mask>
|
||||
</defs>
|
||||
<polygon points="808.0127018922194 433.01270189221935, 58.01270189221947 -1.1368683772161603e-13, 58.01270189221913 866.0254037844386" mask="url(#small-triangle-mask)" fill="white"></polygon>
|
||||
<polyline points="481.2177826491071 333.0127018922194, 134.80762113533166 533.0127018922194" stroke="white" stroke-width="90"></polyline>
|
||||
</svg>
|
||||
</svg>
|
||||
</g>
|
||||
</svg>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 41 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 11 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 136 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 354 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 503 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 186 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 1.7 MiB |
Binary file not shown.
|
After Width: | Height: | Size: 513 KiB |
|
|
@ -0,0 +1,39 @@
|
|||
---
|
||||
id: create-workflow
|
||||
title: How to Create a Sample Workflow
|
||||
sidebar_label: Create Workflow
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
Click on the **Schedule your first workflow** button on the home page to get started. It will take you to the workflow page where you can choose or design your own workflow by doing the following steps:
|
||||
|
||||
### 1. Choose Target Agent
|
||||
|
||||
You have to select a cluster type. There are two types of clusters:
|
||||
|
||||
- Internal Cluster (Self Cluster)
|
||||
- External Clusters that are available on the targets screen.
|
||||
|
||||
After Selecting the cluster, you can continue by clicking on **Select and Continue**.
|
||||
|
||||
```
|
||||
Note:
|
||||
You may have to wait for the cluster to be ready, after which you can move forward by again clicking on “Select and
|
||||
Continue”. Newly created users by the admin can’t schedule a workflow, As non-admin users, you will get a message ‘No Cluster Registered With Your Project ID, Please Wait…’ if you try to create a workflow.
|
||||
```
|
||||
|
||||
### 2. Choose a Workflow
|
||||
|
||||
Next, you need to choose/design the workflow, you will be given a collection of cards grouped as:
|
||||
|
||||
- **Predefined Workflow** : These are some workflows that are provided by default, these are already developed and can be tweaked by adding new environment variables according to the requirements of the user. After Selecting a predefined workflow you can choose to change the name and description of the selected workflow by clicking on “Edit Workflow Name”.
|
||||
|
||||
### 3. Tune Workflow (Optional)
|
||||
|
||||
The next step is tuning the workflow, this section is common for both custom or predefined workflows. You will be given an Editor to check the YAML syntax and also will be able to edit the content in the editor itself. Once the necessary changes have been done, click on Next to move to the next page.
|
||||
|
||||
### 4. Reliability Score (Optional)
|
||||
|
||||
Here, you can set the points for all the experiments present in the selected workflow. These points will affect the Resilience Weights of application to be used with chaos experiment. The weights have been pre-selected as 10 for each test for you. However, you may review and modify the weightage again. The weights are relative to each other which you can change. The successful outcome of each test carries a certain weight. After selecting weights click on the Next Button.
|
||||
|
||||
|
|
@ -0,0 +1,45 @@
|
|||
---
|
||||
id: create
|
||||
title: How to Create a Workflow
|
||||
sidebar_label: Create
|
||||
---
|
||||
|
||||
## Workflow Creation
|
||||
|
||||
---
|
||||
|
||||
Click on the **Schedule your first workflow** button on the home page to get started. It will take you to the workflow page where you can choose or design your own workflow by doing the following steps:
|
||||
|
||||
### 1. Choose Target Cluster
|
||||
|
||||
You have to select a cluster type. There are two types of clusters:
|
||||
|
||||
- Internal Cluster (Self Cluster)
|
||||
- External Clusters that are available on the targets screen.
|
||||
|
||||
After Selecting the cluster, you can continue by clicking on **Select and Continue**.
|
||||
|
||||
```
|
||||
Note:
|
||||
You may have to wait for the cluster to be ready, after which you can move forward by again clicking on “Select and
|
||||
Continue”. Newly created users by the admin can’t schedule a workflow, As non-admin users, you will get a message ‘No Cluster Registered With Your Project ID, Please Wait…’ if you try to create a workflow.
|
||||
```
|
||||
|
||||
### 2. Choose a Workflow
|
||||
|
||||
Next, you need to choose/design the workflow, you will be given a collection of cards grouped as:
|
||||
|
||||
- **Predefined Workflow** : These are some workflows that are provided by default, these are already developed and can be tweaked by adding new environment variables according to the requirements of the user. After Selecting a predefined workflow you can choose to change the name and description of the selected workflow by clicking on “Edit Workflow Name”.
|
||||
|
||||
- **Custom Workflow**: If you are using a Custom workflow you would be given the option to select an experiment, and customize the same. In this section, if you can also **upload** a custom workflow manifest or construct a new one using **experiments from MyHub** with the following steps.
|
||||
Create the workflow by selecting your hub, followed by selecting the experiment and giving the namespace where the workflow has to be scheduled.
|
||||
You can also tune the chaos engine for the experiment by adding new env variables and click add experiment.
|
||||
You can add more experiments and after that, you can also alter the sequence by dragging the selected experiment. Once done, click on **Finish adding experiments** which generates a custom workflow manifest with all the changes that you have added.
|
||||
|
||||
### 3. Tune Workflow
|
||||
|
||||
The next step is tuning the workflow, this section is common for both custom or predefined workflows. You will be given an Editor to check the YAML syntax and also will be able to edit the content in the editor itself. Once the necessary changes have been done, click on Next to move to the next page.
|
||||
|
||||
### 4. Reliability Score
|
||||
|
||||
Here, you can set the points for all the experiments present in the selected workflow. These points will affect the Resilience Weights of application to be used with chaos experiment. The weights have been pre-selected as 10 for each test for you. However, you may review and modify the weightage again. The weights are relative to each other which you can change. The successful outcome of each test carries a certain weight. After selecting weights click on the Next Button.
|
||||
|
|
@ -0,0 +1,43 @@
|
|||
---
|
||||
id: d0-create
|
||||
title: How to create the workflow
|
||||
sidebar_label: Create Workflow
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
Click on the **Schedule your first workflow** button on the home page to get started. It will take you to the workflow page where you can choose or design your own workflow by doing the following steps:
|
||||
|
||||
### 1. Choose Target Cluster
|
||||
|
||||
You have to select a cluster type. There are two types of clusters:
|
||||
|
||||
- Internal Cluster (Self Cluster)
|
||||
- External Clusters that are available on the targets screen.
|
||||
|
||||
After Selecting the cluster, you can continue by clicking on **Select and Continue**.
|
||||
|
||||
```
|
||||
Note:
|
||||
You may have to wait for the cluster to be ready, after which you can move forward by again clicking on “Select and
|
||||
Continue”. Newly created users by the admin can’t schedule a workflow, As non-admin users, you will get a message ‘No Cluster Registered With Your Project ID, Please Wait…’ if you try to create a workflow.
|
||||
```
|
||||
|
||||
### 2. Choose a Workflow
|
||||
|
||||
Next, you need to choose/design the workflow, you will be given a collection of cards grouped as:
|
||||
|
||||
- **Predefined Workflow** : These are some workflows that are provided by default, these are already developed and can be tweaked by adding new environment variables according to the requirements of the user. After Selecting a predefined workflow you can choose to change the name and description of the selected workflow by clicking on “Edit Workflow Name”.
|
||||
|
||||
- **Custom Workflow**: If you are using a Custom workflow you would be given the option to select an experiment, and customize the same. In this section, if you can also **upload** a custom workflow manifest or construct a new one using **experiments from MyHub** with the following steps.
|
||||
Create the workflow by selecting your hub, followed by selecting the experiment and giving the namespace where the workflow has to be scheduled.
|
||||
You can also tune the chaos engine for the experiment by adding new env variables and click add experiment.
|
||||
You can add more experiments and after that, you can also alter the sequence by dragging the selected experiment. Once done, click on **Finish adding experiments** which generates a custom workflow manifest with all the changes that you have added.
|
||||
|
||||
### 3. Tune Workflow
|
||||
|
||||
The next step is tuning the workflow, this section is common for both custom or predefined workflows. You will be given an Editor to check the YAML syntax and also will be able to edit the content in the editor itself. Once the necessary changes have been done, click on Next to move to the next page.
|
||||
|
||||
### 4. Reliability Score
|
||||
|
||||
Here, you can set the points for all the experiments present in the selected workflow. These points will affect the Resilience Weights of application to be used with chaos experiment. The weights have been pre-selected as 10 for each test for you. However, you may review and modify the weightage again. The weights are relative to each other which you can change. The successful outcome of each test carries a certain weight. After selecting weights click on the Next Button.
|
||||
|
|
@ -0,0 +1,47 @@
|
|||
---
|
||||
id: d0-observe
|
||||
title: How to Observe/Track a Workflow
|
||||
sidebar_label: Observe Workflow
|
||||
---
|
||||
|
||||
## Observe a Workflow
|
||||
|
||||
---
|
||||
|
||||
After scheduling a workflow, the user can track the status of the workflow from the Browse Workflow section. The status that is currently displayed are :
|
||||
|
||||
- Failed
|
||||
- Running
|
||||
- Completed
|
||||
|
||||
### 1. Workflow Analysis
|
||||
|
||||
The user can analyze a workflow using two methods:
|
||||
|
||||
#### a. Visualize the workflow graph
|
||||
|
||||
After scheduling a workflow, the user can click on the **Show the workflow** option to see the real-time graph of the workflow.
|
||||
|
||||
<figure>
|
||||
<img src={require('./assets/node-details.png').default} width="800" />
|
||||
<i>Fig 1 : Graph of Kube Proxy Chaos workflow with chaos logs and chaos result of pod-memory-hog experiment.</i>
|
||||
</figure>
|
||||
|
||||
The graph consists of useful information such as :
|
||||
|
||||
- Phase of individual nodes.
|
||||
- Total time taken for the nodes to execute.
|
||||
- Structure of the experiments (Serial or Parallel experiments).
|
||||
- **Logs of individual nodes**: The user can click on the nodes to get the logs of that particular step. If the revert-chaos step is disabled, the complete logs are available which include the runner pod logs and the chaos logs.
|
||||
- **Chaos Results** : Once the experiment completes, the Chaos Results are also available alongside the logs. The Chaos results are directly fetched from the ChaosResult CRD.
|
||||
|
||||
#### b. Analytics from the workflow table
|
||||
|
||||
Once the workflow execution completes, the user can click the **Show the analytics** option which opens up a time series graph of the workflow. This graph can be crucial to analyse the Cron Workflows. Each bar in the graph represents one run of the workflow.
|
||||
On hovering over the bar, the user can view the following information related to the workflow run:
|
||||
|
||||
- Resilience Score
|
||||
- Passed Tests
|
||||
- Failed Tests
|
||||
|
||||
On clicking the bar, a table is shown below the graph which displays all the experiments present in the workflow run and their individual result points. These result points are then used to calculate the **Resilience Score** of the workflow which is displayed at the end of the table.
|
||||
|
|
@ -0,0 +1,22 @@
|
|||
---
|
||||
id: d1-schedule
|
||||
title: How to Schedule a Workflow
|
||||
sidebar_label: Schedule workflow
|
||||
---
|
||||
|
||||
## Workflow Scheduling
|
||||
|
||||
---
|
||||
|
||||
### 1. Scheduling
|
||||
|
||||
After selecting the reliability scores for the tests, the user can now schedule the workflows. The default option in this screen is **Schedule Now**. With this option the user can run a workflow immediately once the workflow creation is completed. The second option available to the user is **Schedule Later**. With this option, the user can schedule the workflow at any time in the future. When this option is selected , the kind of the workflow changes from **Workflow** to **CronWorkflow** with a Cron syntax added in the workflow manifest. The following options are available to the user for scheduling a workflow later:
|
||||
|
||||
- Every Hour
|
||||
- Every Day
|
||||
- Every Week
|
||||
- Every Month
|
||||
|
||||
### 2. Verify and Commit
|
||||
|
||||
After adding the schedules, the user can see the verify and commit screen where all the details of the workflow are present such as the workflow name, the target cluster, workflow manifest, test weights etc. Once the user finishes configuring the workflow, it can be started by selecting the **Finish** button.
|
||||
|
|
@ -0,0 +1,30 @@
|
|||
---
|
||||
id: d1-update
|
||||
title: How to Update a Workflow
|
||||
sidebar_label: Update Workflow
|
||||
---
|
||||
|
||||
## Update a Workflow
|
||||
|
||||
---
|
||||
|
||||
After scheduling a workflow, the user can update the scheduled configuration of the workflow by clicking on Schedules tab.
|
||||
Here the user can view several options like:
|
||||
- Re-Run schedule
|
||||
- Edit Schedule
|
||||
- Delete Schedule
|
||||
- Download Manifest
|
||||
|
||||
### 1. Re-Run Schedule
|
||||
This option allows the user to re-run a workflow with the same configuration as earlier. This option is only available when the workflow is scheduled once.
|
||||
|
||||
### 2. Edit Schedule
|
||||
This option allows the user to edit the schedule configuration of the workflow. The user can either disable the workflow or change the Cron Syntax by changing the scheduled timings.
|
||||
The disable workflow option adds a label **suspend: true** that disables any scheduled workflow.
|
||||
This option is only available if the workflow is a Cron Workflow i.e a recurring workflow.
|
||||
|
||||
### 3. Delete Schedule
|
||||
This option allows the user to delete a scheduled workflow. This is an irreversible process and once the delete option is selected, all the workflow configuration and the schedules are deleted from the portal.
|
||||
|
||||
### 4. Download Manifest
|
||||
This option allows the user to download the manifest of the workflow. The manifest consists of all the configurations that the user has added while workflow creation. The user can then alter changes in this manifest and upload it with the portal to schedule a new workflow.
|
||||
|
|
@ -0,0 +1,33 @@
|
|||
---
|
||||
id: getstarted
|
||||
title: Getting Started with Litmus
|
||||
sidebar_label: Pre-requisites
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
## Pre-requisites
|
||||
|
||||
- Kubernetes 1.15 or later.
|
||||
|
||||
- Persistent volume of 20GB
|
||||
|
||||
Recommend to have a Persistent volume(PV) of 20GB, You can start with 1GB for test purposes as well. This PV is used as persistent storage to store the chaos config and chaos-metrics in the Portal. By default, litmus install would use the default storage class to allocate the PV. Provide this value
|
||||
|
||||
- Helm3 or Kubectl
|
||||
|
||||
## Getting Started
|
||||
|
||||
Running chaos on your application involves the following steps:
|
||||
|
||||
[Install Litmus](litmus-install-cluster-mode)
|
||||
|
||||
[How to Create and Run a Workflow](create-workflow)
|
||||
|
||||
[Observe ChaosResults](observe-workflow)
|
||||
|
||||
<hr/>
|
||||
|
||||
## Join our community
|
||||
|
||||
If you have not joined our community, do join us [here](https://app.slack.com/client/T09NY5SBT/CNXNB0ZTN).
|
||||
|
|
@ -0,0 +1,27 @@
|
|||
---
|
||||
id: gitops
|
||||
title: GitOps
|
||||
sidebar_label: GitOps
|
||||
---
|
||||
|
||||
---
|
||||
GitOps feature in Litmus enables the users to configure a single source of truth for their chaos workflows and experiments, any changes made either to the artifacts stored in the configured git repository or the portal will be synced. This allows creation and execution of workflows directly from git enabling a vast scope of automation in CI/CD pipelines.
|
||||
|
||||
Besides the sync feature GitOps in Litmus provides a way of using Event-Driven Chaos Injection, where target resources(statefulsets, deployments etc.) can be configured to automatically trigger chaos workflows with any changes in the resource spec. Currently the event supported for chaos injection is resource **image change**, but in the future releases a wide variety of policies will be added to configure various events for chaos injection.
|
||||
|
||||
The event driven chaos injection allows Litmus to be integrated with traditional GitOps flow that involves automated deployment of applications or workloads, for example users can now automatically trigger chaos workflows whenever a new release is created for their application and is deployed by a continuous delivery system.
|
||||
|
||||
GitOps is by default disabled for the projects created in Litmus, but it can be enabled and configured from the **GitOps** tab in Litmus portal **Settings**
|
||||
|
||||
## Configuring a Git repository for GitOps
|
||||
|
||||
To configure a git repo the user must provide the Git URL of the repository and the Branch name and the authentication credentials which are of two types:
|
||||
|
||||
#### a. Access Token
|
||||
Personal Access Tokens are used as an alternative to the password for authentication to Git services.
|
||||
|
||||
#### b. SSH Key
|
||||
Just like the Access Token, SSH keys are used for the authentication. These keys come in pairs, a public key that is shared with the Git Services and a private key that is stored in the server.
|
||||
SSH link of the repository should be provided if the user selects this method.
|
||||
|
||||
Once GitOps is enabled, any new workflows created will be stored in the configured repo in the path `litmus/<project-id>/<workflow-name>.yaml`.
|
||||
|
|
@ -0,0 +1,57 @@
|
|||
---
|
||||
id: introduction
|
||||
title: Litmus Introduction
|
||||
sidebar_label: Overview
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
Welcome to the documentation for the **LitmusChaos** project. Litmus is used by devs & SREs alike to create, manage and monitor chaos workflows by extending Kubernetes itself. Find everything that you need for creating and running chaos workflows on and around Kubernetes.
|
||||
|
||||
<img src={require('./assets/litmus-overview.png').default} width="800" />
|
||||
|
||||
Service downtimes are always expensive and they are very difficult to predict and respond to. Certifying if a given change is going to break the services that are functioning well has become increasingly challenging for IT teams. What's more? With the evolution of modern cloud-native micro services these changes have become more dynamic. In such an atmosphere, Chaos Engineering is increasingly becoming the go-to solution within cloud-native ecosystems for asserting the resilience of a service or application at any stage of rolling out changes.
|
||||
|
||||
LitmusChaos is a complete framework to implement chaos engineering within a cloud-native ecosystem. It helps both Developers and SREs automate the chaos experiments at different stages within the DevOps pipeline like development, during CI/CD, & in production.
|
||||
|
||||
- What is a Chaos Experiment?
|
||||
|
||||
Chaos Experiments are fundamental units within the LitmusChaos architecture. Users can choose between readily available chaos experiments or create new ones to construct a required chaos workflow.
|
||||
|
||||
- What is a Chaos Workflow?
|
||||
|
||||
A chaos workflow is much more than a simple chaos experiment. It supports the user in defining the expected result, observing the result, analysing the overall system behaviour, and in the decision-making process if the system needs to be tuned for improving the resilience.
|
||||
LitmusChaos provides necessary infrastructure to develop, use, and manage chaos workflows for a typical development or operations team. Teaming and GitOps features of Litmus help greatly to collaborate the chaos workflows management within the teams or software organisations.
|
||||
|
||||
## Litmus 2.0 Prologue
|
||||
|
||||
The goal of the LitmusChaos project is to create a complete solution to implement chaos engineering at scale, the Kubernetes way! Of course this had to be done incrementally by first creating a toolset for chaos injection and then adding additional features to make it a platform. Litmus 1.x achieved the goal of keeping it completely open source, creating a ChaosHub and the required CRDs, Operators, and Schedulers. With Litmus 1.x, users have a working chaos engineering toolset aligned with the original goals. Over time, the monthly cadence releases added the following features.
|
||||
|
||||
- Chaos experiments become building blocks of a ChaosWorkflow, to allow users to create larger chaos scenarios.
|
||||
|
||||
- A portal to centrally visualize the chaos workflows, get chaos analytics, get the teaming in place for collaborating on chaos workflows.
|
||||
|
||||
- Chaos GitOps for highly scalable automation of chaos workflows. Chaos can now be triggered as a result of a change to an application. This integrates with other CD tools like ArgoCD and FluxCD
|
||||
|
||||
- Chaos Interleaved dashboards. A step toward open observability that is interleaved with chaos incident details.
|
||||
|
||||
_Note:_ Litmus itself is composed of microservices. For 2.0, more microservices have been added and the existing ones will continue to work. Litmus 2.0 is completely backward compatible. No features are deprecated. Migration path is about constructing new artifacts such as ChaosWorkflows that include the current chaos experiments in use by the users.
|
||||
|
||||
Below is a high level comparison between Litmus 1.x and Litmus 2.0 providing a holistic view of the feature additions you get in Litmus 2.0.
|
||||
|
||||
| Litmus 1.x | Litmus 2.0 |
|
||||
| --------------------- | ------------------------------------- |
|
||||
| Experiments | Chaos Workflows |
|
||||
| Per user | Teams (Multi Tenant) |
|
||||
| Per cluster | Per organisation (Cross Cloud) |
|
||||
| Only Public Chaos hub | Public and Private Chaos Hubs |
|
||||
| CLI only | CLI and GUI |
|
||||
| | GitOps |
|
||||
| | Scalability |
|
||||
| | Integrated and Interleaved monitoring |
|
||||
|
||||
## Litmus quick access links
|
||||
|
||||
### [Litmus features](gitops)
|
||||
|
||||
### [Access Litmus API](https://litmuschaos.github.io/litmus/api.html)
|
||||
|
|
@ -0,0 +1,50 @@
|
|||
---
|
||||
id: k8s-support
|
||||
title: Kubernetes Support
|
||||
sidebar_label: Kubernetes Support
|
||||
original_id: k8s-support
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
### Control Plane Resource Requirements
|
||||
|
||||
The Resource requests provided here have been estimated using data gathered manually using different methods -
|
||||
- Using command `kubectl top`
|
||||
- Recommendations from Vertical-Pod-Autoscaler
|
||||
- Recommendations by a great utility Goldilocks.
|
||||
|
||||
These resources are getting monitored continuously and the information below will be updated as the metrics changes.
|
||||
|
||||
<table>
|
||||
<tr>
|
||||
<th>Pod</th>
|
||||
<th>Container</th>
|
||||
<th>CPU(Requested)</th>
|
||||
<th>Memory(Requested)</th>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>litmusportal-frontend</td>
|
||||
<td>litmusportal-frontend</td>
|
||||
<td>25m</td>
|
||||
<td>300M</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>litmusportal-server</td>
|
||||
<td>auth-server</td>
|
||||
<td>15m</td>
|
||||
<td>150M</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>litmusportal-server</td>
|
||||
<td>graphql-server</td>
|
||||
<td>15m</td>
|
||||
<td>150M</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>mongodb</td>
|
||||
<td>mongodb</td>
|
||||
<td>25m</td>
|
||||
<td>300M</td>
|
||||
</tr>
|
||||
</table>
|
||||
|
|
@ -0,0 +1,152 @@
|
|||
---
|
||||
id: litmus-install-cluster-mode
|
||||
title: Litmus Chaos Control Plane (Cluster Mode)
|
||||
sidebar_label: Control Plane (Cluster Mode)
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
<iframe width="560" height="315" src="https://www.youtube.com/embed/rOrKegj5ePI" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
|
||||
|
||||
## Pre-requisites
|
||||
|
||||
- Kubernetes 1.15 or later.
|
||||
|
||||
- Recommend to have a Persistent volume(PV) of 20GB, You can start with 1GB for test purposes as well. This PV is used as persistent storage to store the chaos config and chaos-metrics in the Portal. By default, litmus would use the default storage class to allocate the PV.
|
||||
|
||||
- Helm3 or Kubectl
|
||||
|
||||
## Installation
|
||||
|
||||
Installation of Litmus can be done using either of the below methods
|
||||
|
||||
- [Helm3](#helm_install) chart or
|
||||
- [Kubectl](#kubectl_install) yaml spec file
|
||||
|
||||
### Installation Steps
|
||||
|
||||
The helm chart will install all the required service account configuration and chaos control plane.
|
||||
|
||||
The following steps will help you install litmus via helm.
|
||||
|
||||
#### Step-1: Add the litmus helm repository
|
||||
|
||||
```bash
|
||||
root@demo:~# helm repo add litmuschaos https://litmuschaos.github.io/litmus-helm/
|
||||
"litmuschaos" has been added to your repositories
|
||||
```
|
||||
|
||||
```bash
|
||||
root@demo:~# helm repo list
|
||||
NAME URL
|
||||
litmuschaos https://litmuschaos.github.io/litmus-helm/
|
||||
```
|
||||
|
||||
#### Step-2: Create the litmus namespace
|
||||
|
||||
- The litmus infra components will be placed in this namespace.
|
||||
|
||||
**Note**: The chaos control plane can be placed in any namespace, though it is typically placed in "litmus".
|
||||
|
||||
```bash
|
||||
root@demo:~# kubectl create ns litmus
|
||||
namespace/litmus created
|
||||
```
|
||||
|
||||
#### Step-3: Install the litmus chaos control plane
|
||||
|
||||
```bash
|
||||
root@demo:~# helm install chaos litmuschaos/litmus-2-0-0-beta --namespace=litmus --devel
|
||||
```
|
||||
|
||||
<span style={{color: 'green'}}><b>Expected Output</b></span>
|
||||
|
||||
```
|
||||
NAME: chaos
|
||||
LAST DEPLOYED: Tue Jun 15 19:20:09 2021
|
||||
NAMESPACE: litmus
|
||||
STATUS: deployed
|
||||
REVISION: 1
|
||||
TEST SUITE: None
|
||||
NOTES:
|
||||
Thank you for installing litmus-2-0-0-beta 😀
|
||||
|
||||
Your release is named chaos and its installed to namespace: litmus.
|
||||
|
||||
Visit https://docs.litmuschaos.io/docs/getstarted/ to find more info.
|
||||
|
||||
```
|
||||
|
||||
> **Note:** Litmus uses Kubernetes CRDs to define chaos intent. Helm3 handles CRDs better than Helm2. Before you start running a chaos experiment, verify if Litmus is installed correctly.
|
||||
|
||||
### <a name="kubectl_install"> </a>**Install Litmus using kubectl **
|
||||
|
||||
#### **Create a Litmus namespace in Kubernetes**
|
||||
|
||||
```bash
|
||||
kubectl create ns litmus
|
||||
```
|
||||
|
||||
#### **Install Litmus**
|
||||
|
||||
Applying the manifest file will install all the required service account configuration and chaos control plane.
|
||||
|
||||
```bash
|
||||
kubectl apply -f https://litmuschaos.github.io/litmus/2.0.0-Beta/litmus-2.0.0-Beta.yaml
|
||||
```
|
||||
|
||||
## **Verify your installation**
|
||||
|
||||
**Verify if the frontend, server, and database pod are running**
|
||||
|
||||
```bash
|
||||
kubectl get pods -n litmus
|
||||
```
|
||||
|
||||
<span style={{color: 'green'}}><b>Expected Output</b></span>
|
||||
|
||||
```bash
|
||||
chaos-litmus-portal-frontend-ff8b554dc-q5rl4 1/1 Running 0 2m6s
|
||||
chaos-litmus-portal-mongo-6764cfdd59-c9r56 1/1 Running 0 2m6s
|
||||
chaos-litmus-portal-server-5ffbccbfff-dknv8 2/2 Running 0 2m6s
|
||||
```
|
||||
|
||||
## **Setup the Portal**
|
||||
|
||||
To setup and login to Litmus Portal expand the available services just created and copy the `PORT` of the `litmusportal-frontend-service` service
|
||||
|
||||
```bash
|
||||
kubectl get svc -n litmus
|
||||
```
|
||||
|
||||
<span style={{color: 'green'}}><b>Expected Output</b></span>
|
||||
|
||||
```bash
|
||||
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
|
||||
chaos-litmus-portal-mongo ClusterIP 10.104.107.117 <none> 27017/TCP 2m
|
||||
litmusportal-frontend-service NodePort 10.101.81.70 <none> 9091:30385/TCP 2m
|
||||
litmusportal-server-service NodePort 10.108.151.79 <none> 9002:32456/TCP,9003:31160/TCP 2m
|
||||
```
|
||||
|
||||
> **Note**: In this case, the PORT for `litmusportal-frontend-service` is `30385`. Yours will be different.
|
||||
|
||||
### **Accessing the Portal**
|
||||
|
||||
Once you have the PORT copied in your clipboard, simply use your NodeIP and PORT in this manner `<NodeIP>:<PORT>` to access the portal.
|
||||
|
||||
For example:
|
||||
|
||||
```yaml
|
||||
http://172.17.0.3:30385/
|
||||
```
|
||||
|
||||
> Where `172.17.0.3` is my NodeIP and `30385` is the frontend service PORT. You can also use a LoadBalancer if not NodePort, the only change would be to provide a `<LoadBalancerIP>:<PORT>`
|
||||
|
||||
You should be able to see the Login Page of Litmus. The **default credentials** are
|
||||
|
||||
```yaml
|
||||
Username: admin
|
||||
Password: litmus
|
||||
```
|
||||
|
||||
<img src={require('./assets/login.png').default} width="800" />
|
||||
|
|
@ -0,0 +1,236 @@
|
|||
---
|
||||
id: litmus-install-namespace-mode
|
||||
title: Litmus Chaos Control Plane (Namespace Mode)
|
||||
sidebar_label: Control Plane (Namespace Mode)
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
<iframe width="560" height="315" src="https://www.youtube.com/embed/rOrKegj5ePI" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
|
||||
|
||||
## Pre-requisites
|
||||
|
||||
- Kubernetes 1.15 or later.
|
||||
|
||||
- Recommend to have a Persistent volume(PV) of 20GB, You can start with 1GB for test purposes as well. This PV is used as persistent storage to store the chaos config and chaos-metrics in the Portal. By default, litmus would use the default storage class to allocate the PV.
|
||||
|
||||
- Helm3 or Kubectl
|
||||
|
||||
## Installation
|
||||
|
||||
Installation of Litmus can be done using either of the below methods
|
||||
|
||||
- [Helm3](#helm_install) chart or
|
||||
- [Kubectl](#kubectl_install) yaml spec file
|
||||
|
||||
### <a name="kubectl_install"> </a>**Install Litmus using Helm **
|
||||
|
||||
The helm chart will install all the required service account configuration and chaos control plane.
|
||||
|
||||
The following steps will help you install litmus via helm.
|
||||
|
||||
#### Step-1: Add the litmus helm repository
|
||||
|
||||
```bash
|
||||
root@demo:~# helm repo add litmuschaos https://litmuschaos.github.io/litmus-helm/
|
||||
"litmuschaos" has been added to your repositories
|
||||
```
|
||||
|
||||
```bash
|
||||
root@demo:~# helm repo list
|
||||
NAME URL
|
||||
litmuschaos https://litmuschaos.github.io/litmus-helm/
|
||||
```
|
||||
|
||||
#### Step-2: Create the litmus namespace
|
||||
|
||||
- The litmus infra components will be placed in this namespace.
|
||||
|
||||
**Note**: The chaos control plane can be placed in any namespace, though it is typically placed in "litmus".
|
||||
|
||||
```bash
|
||||
root@demo:~# kubectl create ns litmus
|
||||
namespace/litmus created
|
||||
```
|
||||
|
||||
#### Step-3: Install the litmus chaos control plane
|
||||
|
||||
```bash
|
||||
root@demo:~# helm install chaos litmuschaos/litmus-2-0-0-beta --namespace=litmus --devel --set portalScope=namespace
|
||||
```
|
||||
|
||||
<span style={{color: 'green'}}><b>Expected Output</b></span>
|
||||
|
||||
```bash
|
||||
NAME: chaos
|
||||
LAST DEPLOYED: Tue Jun 15 19:20:09 2021
|
||||
NAMESPACE: litmus
|
||||
STATUS: deployed
|
||||
REVISION: 1
|
||||
TEST SUITE: None
|
||||
NOTES:
|
||||
Thank you for installing litmus-2-0-0-beta 😀
|
||||
|
||||
Your release is named chaos and its installed to namespace: litmus.
|
||||
|
||||
Visit https://docs.litmuschaos.io/docs/getstarted/ to find more info.
|
||||
|
||||
```
|
||||
|
||||
> **Note:** Litmus uses Kubernetes CRDs to define chaos intent. Helm3 handles CRDs better than Helm2. Before you start running a chaos experiment, verify if Litmus is installed correctly.
|
||||
|
||||
- The cluster-admin or an equivalent user with the right permissions are required to install them CRDs upfront. To apply LitmusCRDs:
|
||||
|
||||
```bash
|
||||
kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/master/litmus-portal/litmus-portal-crds.yml
|
||||
```
|
||||
|
||||
Output:
|
||||
|
||||
```bash
|
||||
customresourcedefinition.apiextensions.k8s.io/clusterworkflowtemplates.argoproj.io created
|
||||
customresourcedefinition.apiextensions.k8s.io/cronworkflows.argoproj.io created
|
||||
customresourcedefinition.apiextensions.k8s.io/workflows.argoproj.io created
|
||||
customresourcedefinition.apiextensions.k8s.io/workflowtemplates.argoproj.io created
|
||||
customresourcedefinition.apiextensions.k8s.io/chaosengines.litmuschaos.io created
|
||||
customresourcedefinition.apiextensions.k8s.io/chaosexperiments.litmuschaos.io created
|
||||
customresourcedefinition.apiextensions.k8s.io/chaosresults.litmuschaos.io created
|
||||
customresourcedefinition.apiextensions.k8s.io/eventtrackerpolicies.eventtracker.litmuschaos.io created
|
||||
```
|
||||
|
||||
### <a name="kubectl_install"> </a>**Install Litmus using kubectl **
|
||||
|
||||
#### **Install Litmus**
|
||||
|
||||
- Set the namespace on which you want to install litmus.
|
||||
|
||||
```bash
|
||||
export LITMUS_PORTAL_NAMESPACE="<namespace>"
|
||||
kubectl get ns ${LITMUS_PORTAL_NAMESPACE}
|
||||
# If the namespace is not already present
|
||||
# then create the target namespace
|
||||
kubectl create ns ${LITMUS_PORTAL_NAMESPACE}
|
||||
```
|
||||
|
||||
- The cluster-admin or an equivalent user with the right permissions are required to install them CRDs upfront. To apply LitmusCRDs:
|
||||
|
||||
```bash
|
||||
kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/master/litmus-portal/litmus-portal-crds.yml
|
||||
```
|
||||
|
||||
Output:
|
||||
|
||||
```bash
|
||||
customresourcedefinition.apiextensions.k8s.io/clusterworkflowtemplates.argoproj.io created
|
||||
customresourcedefinition.apiextensions.k8s.io/cronworkflows.argoproj.io created
|
||||
customresourcedefinition.apiextensions.k8s.io/workflows.argoproj.io created
|
||||
customresourcedefinition.apiextensions.k8s.io/workflowtemplates.argoproj.io created
|
||||
customresourcedefinition.apiextensions.k8s.io/chaosengines.litmuschaos.io created
|
||||
customresourcedefinition.apiextensions.k8s.io/chaosexperiments.litmuschaos.io created
|
||||
customresourcedefinition.apiextensions.k8s.io/chaosresults.litmuschaos.io created
|
||||
customresourcedefinition.apiextensions.k8s.io/eventtrackerpolicies.eventtracker.litmuschaos.io created
|
||||
```
|
||||
|
||||
- Replace namespace with the target namespace.
|
||||
|
||||
```bash
|
||||
export LITMUS_PORTAL_NAMESPACE="<namespace>"
|
||||
curl https://raw.githubusercontent.com/litmuschaos/litmus/master/docs/2.0.0-Beta/litmus-namespaced-2.0.0-Beta.yaml --output litmus-portal-namespaced-K8s-template.yml
|
||||
envsubst < litmus-portal-namespaced-K8s-template.yml >
|
||||
${LITMUS_PORTAL_NAMESPACE}-ns-scoped-litmus-portal-manifest.yml
|
||||
kubectl apply -f ${LITMUS_PORTAL_NAMESPACE}-ns-scoped-litmus-portal-manifest.yml -n ${LITMUS_PORTAL_NAMESPACE}
|
||||
```
|
||||
|
||||
Output:
|
||||
|
||||
```bash
|
||||
configmap/litmus-portal-admin-config created
|
||||
deployment.apps/litmusportal-frontend created
|
||||
service/litmusportal-frontend-service created
|
||||
serviceaccount/litmus-server-account created
|
||||
role.rbac.authorization.k8s.io/litmus-server created
|
||||
rolebinding.rbac.authorization.k8s.io/litmus-server-rb created
|
||||
deployment.apps/litmusportal-server created
|
||||
service/litmusportal-server-service created
|
||||
statefulset.apps/mongo created
|
||||
service/mongo-service created
|
||||
```
|
||||
|
||||
## **Verify your installation**
|
||||
|
||||
**Verify if the frontend, server, and database pods are running**
|
||||
|
||||
- Check the litmus CRDs:
|
||||
|
||||
```bash
|
||||
$ kubectl get crds | grep litmus
|
||||
chaosengines.litmuschaos.io 2021-03-24T06:52:54Z
|
||||
chaosexperiments.litmuschaos.io 2021-03-24T06:52:54Z
|
||||
chaosresults.litmuschaos.io 2021-03-24T06:52:55Z
|
||||
eventtrackerpolicies.eventtracker.litmuschaos.io 2021-03-24T06:52:55Z
|
||||
|
||||
```
|
||||
|
||||
- Check the pods in litmus namespace:
|
||||
|
||||
```bash
|
||||
$ kubectl get pods -n litmus
|
||||
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
litmusportal-frontend-97c8bf86b-mx89w 1/1 Running 2 6m24s
|
||||
litmusportal-server-5cfbfc88cc-m6c5j 2/2 Running 2 6m19s
|
||||
mongo-0 1/1 Running 0 6m16s
|
||||
```
|
||||
|
||||
- Check the services running in litmus namespace:
|
||||
|
||||
```bash
|
||||
$ kubectl get svc -n litmus
|
||||
|
||||
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
|
||||
litmusportal-frontend-service NodePort 10.100.105.154 <none> 9091:30229/TCP 7m14s
|
||||
litmusportal-server-service NodePort 10.100.150.175 <none> 9002:30479/TCP,9003:31949/TCP 7m8s
|
||||
mongo-service ClusterIP 10.100.226.179 <none> 27017/TCP 7m6s
|
||||
```
|
||||
|
||||
<br />
|
||||
|
||||
## **Setup the Portal**
|
||||
|
||||
To setup and login to Litmus Portal expand the available services just created and copy the `PORT` of the `litmusportal-frontend-service` service
|
||||
|
||||
```bash
|
||||
kubectl get svc -n litmus
|
||||
```
|
||||
|
||||
<span style={{color: 'green'}}><b>Expected Output</b></span>
|
||||
|
||||
```bash
|
||||
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
|
||||
chaos-litmus-portal-mongo ClusterIP 10.104.107.117 <none> 27017/TCP 2m
|
||||
litmusportal-frontend-service NodePort 10.101.81.70 <none> 9091:30385/TCP 2m
|
||||
litmusportal-server-service NodePort 10.108.151.79 <none> 9002:32456/TCP,9003:31160/TCP 2m
|
||||
```
|
||||
|
||||
> **Note**: In this case, the PORT for `litmusportal-frontend-service` is `30385`. Yours will be different.
|
||||
|
||||
### **Accessing the Portal**
|
||||
|
||||
Once you have the PORT copied in your clipboard, simply use your NodeIP and PORT in this manner `<NodeIP>:<PORT>` to access the portal.
|
||||
|
||||
For example:
|
||||
|
||||
```yaml
|
||||
http://172.17.0.3:30385/
|
||||
```
|
||||
|
||||
> Where `172.17.0.3` is my NodeIP and `30385` is the frontend service PORT. You can also use a LoadBalancer if not NodePort, the only change would be to provide a `<LoadBalancerIP>:<PORT>`
|
||||
|
||||
You should be able to see the Login Page of Litmus. The **default credentials** are
|
||||
|
||||
```yaml
|
||||
Username: admin
|
||||
Password: litmus
|
||||
```
|
||||
|
||||
<img src={require('./assets/login.png').default} width="800" />
|
||||
|
|
@ -0,0 +1,129 @@
|
|||
---
|
||||
id: litmus-with-ingress
|
||||
title: Install LitmusPortal with Ingress
|
||||
sidebar_label: Litmus with Ingress
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
### Install LitmusPortal with Ingress
|
||||
|
||||
With Litmus-2.0.0-Beta3, LitmusPortal can be installed with ingress.
|
||||
In the following doc, we will use the Nginx ingress controller for ingress setup.
|
||||
|
||||
1. Install the litmus chaos control plane
|
||||
|
||||
```bash
|
||||
kubectl apply -f https://litmuschaos.github.io/litmus/2.0.0-Beta/litmus-2.0.0-Beta.yaml
|
||||
```
|
||||
|
||||
2. By default, the service type is NodePort. For Ingress, we need to change the service type to ClusterIP in the following services.
|
||||
* litmusportal-frontend-service
|
||||
* litmusportal-server-service
|
||||
|
||||
|
||||
3. Install Nginx Ingress Controller along with Kubernetes RBAC roles and bindings, please refer [here](https://kubernetes.github.io/ingress-nginx/deploy/#installation-guide)
|
||||
|
||||
**Note:**
|
||||
* If you're changing ingress name from **litmus-ingress** to a different name, make sure to update the **INGRESS_NAME** environment variable in the litmusportal-server deployment
|
||||
* Set the environment variable **INGRESS** as true in the litmusportal-server deployment.
|
||||
|
||||
#### With HTTP
|
||||
|
||||
4. Sample litmus ingress manifest With HTTP
|
||||
|
||||
```yaml
|
||||
apiVersion: extensions/v1beta1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
annotations:
|
||||
kubernetes.io/ingress.class: nginx
|
||||
nginx.ingress.kubernetes.io/rewrite-target: /$1
|
||||
name: litmus-ingress
|
||||
spec:
|
||||
rules:
|
||||
- host: "<HOST-NAME>"
|
||||
http:
|
||||
paths:
|
||||
- backend:
|
||||
serviceName: litmusportal-frontend-service
|
||||
servicePort: 9091
|
||||
path: /(.*)
|
||||
pathType: ImplementationSpecific
|
||||
- backend:
|
||||
serviceName: litmusportal-server-service
|
||||
servicePort: 9002
|
||||
path: /backend/(.*)
|
||||
pathType: ImplementationSpecific
|
||||
```
|
||||
|
||||
```bash
|
||||
kubectl apply -f <litmus_ingress_manifest> -n <PORTAL_NAMESPACE>
|
||||
```
|
||||
|
||||
#### With HTTPS
|
||||
|
||||
4. Install CertManager
|
||||
|
||||
```bash
|
||||
kubectl create namespace cert-manager
|
||||
helm repo add jetstack https://charts.jetstack.io
|
||||
helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --version v1.3.0 --set installCRDs=true
|
||||
```
|
||||
|
||||
5. Install LetsEncrypt Cluster Issuer
|
||||
|
||||
```yaml
|
||||
apiVersion: cert-manager.io/v1alpha2
|
||||
kind: ClusterIssuer
|
||||
metadata:
|
||||
name: letsencrypt
|
||||
spec:
|
||||
acme:
|
||||
server: https://acme-v02.api.letsencrypt.org/directory
|
||||
email: "your@mail.com"
|
||||
privateKeySecretRef:
|
||||
name: letsencrypt
|
||||
solvers:
|
||||
- http01:
|
||||
ingress:
|
||||
class: nginx
|
||||
```
|
||||
|
||||
6. Sample Litmus Portal Ingress Manifest with HTTPS
|
||||
|
||||
```yaml
|
||||
apiVersion: extensions/v1beta1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
annotations:
|
||||
cert-manager.io/cluster-issuer: letsencrypt
|
||||
kubernetes.io/ingress.class: nginx
|
||||
nginx.ingress.kubernetes.io/rewrite-target: /$1
|
||||
labels:
|
||||
component: litmusportal-frontend
|
||||
name: litmusportal-ingress
|
||||
namespace: litmus
|
||||
spec:
|
||||
rules:
|
||||
- host: "<HOST-NAME>"
|
||||
http:
|
||||
paths:
|
||||
- backend:
|
||||
serviceName: litmusportal-frontend-service
|
||||
servicePort: 9091
|
||||
path: /(.*)
|
||||
pathType: ImplementationSpecific
|
||||
- backend:
|
||||
serviceName: litmusportal-server-service
|
||||
servicePort: 9002
|
||||
path: /backend/(.*)
|
||||
pathType: ImplementationSpecific
|
||||
tls:
|
||||
- hosts:
|
||||
- "<HOST-NAME>"
|
||||
secretName: litmuspreview-tls-secret
|
||||
```
|
||||
```bash
|
||||
kubectl apply -f <litmus_ingress_manifest> -n <PORTAL_NAMESPACE>
|
||||
```
|
||||
|
|
@ -0,0 +1,145 @@
|
|||
---
|
||||
id: litmus-psp
|
||||
title: Using Pod Security Policies with Litmus
|
||||
sidebar_label: Chaos Pod Security Policies
|
||||
original_id: litmus-psp
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
While working in environments (clusters) that have restrictive security policies, the default litmuschaos experiment execution procedure may be inhibited. This is mainly due to the fact that the experiment pods running the chaos injection tasks run with a root user. This, in turn, is necessitated due to the mounting of container runtime-specific socket files from the Kubernetes nodes in order to invoke runtime APIs. While this is not needed for all experiments (a considerable number of them use purely the K8s API), those involving injection of chaos processes into the network/process namespaces of other containers have this requirement
|
||||
(ex: netem, stress).
|
||||
|
||||
The restrictive policies are often enforced via [pod security policies](https://kubernetes.io/docs/concepts/policy/pod-security-policy/) (PSP) today, with organizations opting for the default ["restricted"](https://kubernetes.io/docs/concepts/policy/pod-security-policy/#example-policies) policy.
|
||||
|
||||
## Applying Pod Security Policies to Litmus Chaos Pods
|
||||
|
||||
- To run the litmus pods with operating characteristics described above, first create a custom PodSecurityPolicy that allows the same:
|
||||
|
||||
[embedmd]: # "https://raw.githubusercontent.com/litmuschaos/chaos-charts/master/pod-security-policy/psp-litmus.yaml yaml"
|
||||
|
||||
```yaml
|
||||
apiVersion: policy/v1beta1
|
||||
kind: PodSecurityPolicy
|
||||
metadata:
|
||||
name: litmus
|
||||
annotations:
|
||||
seccomp.security.alpha.kubernetes.io/allowedProfileNames: "*"
|
||||
spec:
|
||||
privileged: true
|
||||
# Required to prevent escalations to root.
|
||||
allowPrivilegeEscalation: true
|
||||
# Allow core volume types.
|
||||
volumes:
|
||||
- "configMap"
|
||||
- "emptyDir"
|
||||
- "projected"
|
||||
- "secret"
|
||||
- "downwardAPI"
|
||||
# Assume that persistentVolumes set up by the cluster admin are safe to use.
|
||||
- "persistentVolumeClaim"
|
||||
allowedHostPaths:
|
||||
# substitutes this path with an appropriate socket path
|
||||
# ex: '/var/run/docker.sock', '/run/containerd/containerd.sock', '/run/crio/crio.sock'
|
||||
- pathPrefix: "/var/run/docker.sock"
|
||||
# substitutes this path with an appropriate container path
|
||||
# ex: '/var/lib/docker/containers', '/var/lib/containerd/io.containerd.runtime.v1.linux/k8s.io', '/var/lib/containers/storage/overlay/'
|
||||
- pathPrefix: "/var/lib/docker/containers"
|
||||
allowedCapabilities:
|
||||
- "NET_ADMIN"
|
||||
- "SYS_ADMIN"
|
||||
hostNetwork: false
|
||||
hostIPC: false
|
||||
hostPID: true
|
||||
runAsUser:
|
||||
rule: "RunAsAny"
|
||||
seLinux:
|
||||
# This policy assumes the nodes are using AppArmor rather than SELinux.
|
||||
rule: "RunAsAny"
|
||||
supplementalGroups:
|
||||
rule: "MustRunAs"
|
||||
ranges:
|
||||
# Forbid adding the root group.
|
||||
- min: 1
|
||||
max: 65535
|
||||
fsGroup:
|
||||
rule: "MustRunAs"
|
||||
ranges:
|
||||
# Forbid adding the root group.
|
||||
- min: 1
|
||||
max: 65535
|
||||
readOnlyRootFilesystem: false
|
||||
```
|
||||
|
||||
**Note**: This PodSecurityPolicy is a sample configuration that works for a majority of the use-cases. It is left to the user's discretion to modify it based on the environment. For example, if the experiment doesn't need the socket file to be mounted, `allowedHostPaths` can be excluded from the psp spec. On the other hand, in case of CRI-O runtime, network-chaos tests need the chaos pods executed in privileged mode. It is also possible that different PSP configs are used in different namespaces based on ChaosExperiments installed/executed in them.
|
||||
|
||||
- Subscribe to the created PSP in the experiment RBAC (or in the [admin-mode](https://docs.litmuschaos.io/docs/admin-mode/#prepare-rbac-manifest) rbac, as applicable). For example, the pod-delete experiment rbac instrumented with the PSP is shown below:
|
||||
|
||||
[embedmd]: # "https://raw.githubusercontent.com/litmuschaos/chaos-charts/master/charts/generic/pod-delete/rbac-psp.yaml yaml"
|
||||
|
||||
```yaml
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: pod-delete-sa
|
||||
namespace: default
|
||||
labels:
|
||||
name: pod-delete-sa
|
||||
app.kubernetes.io/part-of: litmus
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: Role
|
||||
metadata:
|
||||
name: pod-delete-sa
|
||||
namespace: default
|
||||
labels:
|
||||
name: pod-delete-sa
|
||||
app.kubernetes.io/part-of: litmus
|
||||
rules:
|
||||
- apiGroups: [""]
|
||||
resources: ["pods", "events"]
|
||||
verbs:
|
||||
["create", "list", "get", "patch", "update", "delete", "deletecollection"]
|
||||
- apiGroups: [""]
|
||||
resources: ["pods/exec", "pods/log", "replicationcontrollers"]
|
||||
verbs: ["create", "list", "get"]
|
||||
- apiGroups: ["batch"]
|
||||
resources: ["jobs"]
|
||||
verbs: ["create", "list", "get", "delete", "deletecollection"]
|
||||
- apiGroups: ["apps"]
|
||||
resources: ["deployments", "statefulsets", "daemonsets", "replicasets"]
|
||||
verbs: ["list", "get"]
|
||||
- apiGroups: ["apps.openshift.io"]
|
||||
resources: ["deploymentconfigs"]
|
||||
verbs: ["list", "get"]
|
||||
- apiGroups: ["argoproj.io"]
|
||||
resources: ["rollouts"]
|
||||
verbs: ["list", "get"]
|
||||
- apiGroups: ["litmuschaos.io"]
|
||||
resources: ["chaosengines", "chaosexperiments", "chaosresults"]
|
||||
verbs: ["create", "list", "get", "patch", "update"]
|
||||
- apiGroups: ["policy"]
|
||||
resources: ["podsecuritypolicies"]
|
||||
verbs: ["use"]
|
||||
resourceNames: ["litmus"]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata:
|
||||
name: pod-delete-sa
|
||||
namespace: default
|
||||
labels:
|
||||
name: pod-delete-sa
|
||||
app.kubernetes.io/part-of: litmus
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: Role
|
||||
name: pod-delete-sa
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: pod-delete-sa
|
||||
namespace: default
|
||||
```
|
||||
|
||||
- Execute the ChaosEngine and verify that the litmus experiment pods are created successfully.
|
||||
|
|
@ -0,0 +1,24 @@
|
|||
---
|
||||
id: litmus-uninstall
|
||||
title: Litmus Chaos Control Plane Uninstall
|
||||
sidebar_label: Control Plane
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
### **Uninstallation of Litmus Control Plane using Helm**
|
||||
|
||||
The helm chart uninstall the control plane components
|
||||
|
||||
```bash
|
||||
helm uninstall litmuschaos --namespace litmus
|
||||
kubectl delete ns litmus
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### **Uninstallation of Litmus Control Plane using kubectl**
|
||||
|
||||
```bash
|
||||
kubectl delete ns litmus
|
||||
```
|
||||
|
|
@ -0,0 +1,50 @@
|
|||
---
|
||||
id: myhub
|
||||
title: MyHubs
|
||||
sidebar_label: MyHubs
|
||||
---
|
||||
|
||||
---
|
||||
MyHubs allows a user to orchestrate workflows from the Public **[ChaosHub](http://hub.litmuschaos.io/)** or an alternate source for the Experiments (basically, a **[fork](https://github.com/litmuschaos/chaos-charts)** of the public source with custom experiments).
|
||||
|
||||
With this feature, a user can construct workflows by selecting, tuning and sequencing different experiments from their connected My Hubs.
|
||||
|
||||
The user can make changes in their forked repositories and sync it with the Portal to get the latest changes from the fork.
|
||||
|
||||
## Connecting a Git repository using MyHubs
|
||||
### 1. Connect a Public Git Repository
|
||||
|
||||
The user can connect to a public Git repository by simply providing the following details:
|
||||
- Hub Name
|
||||
- Git URL of the forked repository
|
||||
- Branch Name
|
||||
|
||||
### 2. Connect a Private Git Repository
|
||||
|
||||
To add a Private Hub, the user should provide the Hub name, Git URL of the forked repository and the Branch name similar to that of Public Hub and the repository can be connected by 2 methods:
|
||||
|
||||
#### a. Access Token
|
||||
Personal Access Tokens are used as an alternative to the password for authentication to Git services.
|
||||
|
||||
#### b. SSH Key
|
||||
Just like the Access Token , SSH keys are used for the authentication. These keys come in pairs, a public key that is shared with the Git Services and a private key that is stored with the user.
|
||||
SSH link of the repository should be provided if the user selects this method.
|
||||
|
||||
## Syncing a MyHub
|
||||
If some changes are made into the git repository, the user can reflect those changes in the hub by selecting the **Refresh Hub** option from the MyHub card.
|
||||
|
||||
## Editing a MyHub
|
||||
To make changes in a hub like changing the name, branch, access token etc, the user can select the **Edit Hub** option from the MyHub card.
|
||||
|
||||
## Experiments in a MyHub
|
||||
### 1. View the chart
|
||||
After connecting a hub, the user can view the different charts and the experiment. The charts are sorted according to different categories like generic, cassandra, kube-components etc.
|
||||
|
||||
### 2. View the experiment details
|
||||
The user can select one of the chart and can examine the experiment details.
|
||||
The experiment page consists of all the important details like the description of the experiment, a tutorial video, the maintainer of the experiment etc.
|
||||
The user can also find experiment yaml link, RBAC link and the Chaos Engine yaml link of the experiment.
|
||||
These yaml links are required for the creation of Custom Chaos Workflows.
|
||||
|
||||
## Disconnect a MyHub
|
||||
To remove a MyHub from a project, the user can select the **Disconnect Hub** option from the MyHub card.
|
||||
|
|
@ -0,0 +1,41 @@
|
|||
---
|
||||
id: observe-workflow
|
||||
title: How to Observe/Track a Workflow
|
||||
sidebar_label: Observe Workflow
|
||||
---
|
||||
|
||||
## Observe a Workflow
|
||||
|
||||
---
|
||||
|
||||
After scheduling a workflow, the user can track the status of the workflow from the Browse Workflow section. The status that is currently displayed are :
|
||||
|
||||
- Failed
|
||||
- Running
|
||||
- Completed
|
||||
|
||||
### 1. Workflow Analysis
|
||||
|
||||
The user can analyze a workflow using two methods:
|
||||
|
||||
#### a. Visualize the workflow graph
|
||||
|
||||
After scheduling a workflow, the user can click on the **Show the workflow** option to see the real-time graph of the workflow.
|
||||
|
||||
<figure>
|
||||
<img src={require('./assets/argo-graph.png').default} width="800" />
|
||||
<i>Fig 1 : Graph of Kube Proxy Chaos workflow.</i>
|
||||
</figure>
|
||||
|
||||
The graph consists of useful information such as :
|
||||
|
||||
- Phase of individual nodes.
|
||||
- Total time taken for the nodes to execute.
|
||||
- Structure of the experiments (Serial or Parallel experiments).
|
||||
- **Logs of individual nodes**: The user can click on the nodes to get the logs of that particular step. If the revert-chaos step is disabled, the complete logs are available which include the runner pod logs and the chaos logs.
|
||||
- **Chaos Results** : Once the experiment completes, the Chaos Results are also available alongside the logs. The Chaos results are directly fetched from the ChaosResult CRD.
|
||||
|
||||
<figure>
|
||||
<img src={require('./assets/node-details.png').default} width="1000" />
|
||||
<i>Fig 2 : Graph of Kube Proxy Chaos workflow with chaos logs and chaos result of pod-memory-hog experiment.</i>
|
||||
</figure>
|
||||
|
|
@ -0,0 +1,47 @@
|
|||
---
|
||||
id: observe
|
||||
title: How to Observe/Track a Workflow
|
||||
sidebar_label: Observe
|
||||
---
|
||||
|
||||
## Observe a Workflow
|
||||
|
||||
---
|
||||
|
||||
After scheduling a workflow, the user can track the status of the workflow from the Browse Workflow section. The status that is currently displayed are :
|
||||
|
||||
- Failed
|
||||
- Running
|
||||
- Completed
|
||||
|
||||
### 1. Workflow Analysis
|
||||
|
||||
The user can analyze a workflow using two methods:
|
||||
|
||||
#### a. Visualize the workflow graph
|
||||
|
||||
After scheduling a workflow, the user can click on the **Show the workflow** option to see the real-time graph of the workflow.
|
||||
|
||||
<figure>
|
||||
<img src={require('./assets/node-details.png').default} width="800" />
|
||||
<i>Fig 1 : Graph of Kube Proxy Chaos workflow with chaos logs and chaos result of pod-memory-hog experiment.</i>
|
||||
</figure>
|
||||
|
||||
The graph consists of useful information such as :
|
||||
|
||||
- Phase of individual nodes.
|
||||
- Total time taken for the nodes to execute.
|
||||
- Structure of the experiments (Serial or Parallel experiments).
|
||||
- **Logs of individual nodes**: The user can click on the nodes to get the logs of that particular step. If the revert-chaos step is disabled, the complete logs are available which include the runner pod logs and the chaos logs.
|
||||
- **Chaos Results** : Once the experiment completes, the Chaos Results are also available alongside the logs. The Chaos results are directly fetched from the ChaosResult CRD.
|
||||
|
||||
#### b. Analytics from the workflow table
|
||||
|
||||
Once the workflow execution completes, the user can click the **Show the analytics** option which opens up a time series graph of the workflow. This graph can be crucial to analyse the Cron Workflows. Each bar in the graph represents one run of the workflow.
|
||||
On hovering over the bar, the user can view the following information related to the workflow run:
|
||||
|
||||
- Resilience Score
|
||||
- Passed Tests
|
||||
- Failed Tests
|
||||
|
||||
On clicking the bar, a table is shown below the graph which displays all the experiments present in the workflow run and their individual result points. These result points are then used to calculate the **Resilience Score** of the workflow which is displayed at the end of the table.
|
||||
|
|
@ -0,0 +1,257 @@
|
|||
---
|
||||
id: probes
|
||||
title: Declarative Approach to Chaos Hypothesis using Litmus Probes
|
||||
sidebar_label: Probes
|
||||
---
|
||||
|
||||
---
|
||||
## Litmus Probes
|
||||
|
||||
Litmus probes are pluggable checks that can be defined within the ChaosEngine for any chaos experiment. The experiment pods execute these checks based on the mode they are defined in & factor their success as necessary conditions in determining the verdict of the experiment (along with the standard “in-built” checks).
|
||||
|
||||
Litmus currently supports four types of probes:
|
||||
|
||||
* **httpProbe:** To query health/downstream URIs
|
||||
* **cmdProbe:** To execute any user-desired health-check function implemented as a shell command
|
||||
* **k8sProbe:** To perform CRUD operations against native & custom Kubernetes resources
|
||||
* **promProbe:** To execute promql queries and match prometheus metrics for specific criteria
|
||||
|
||||
These probes can be used in isolation or in several combinations to achieve the desired checks. While the `httpProbe` & `k8sProbe` are fully declarative in the way they are conceived, the `cmdProbe` expects the user to provide a shell command to implement checks that are highly specific to the application use case. `promProbe` expects the user to provide a promql query along with Prometheus service endpoints to check for specific criteria.
|
||||
|
||||
The probes can be set up to run in different modes:
|
||||
|
||||
* **SoT:** Executed at the Start of Test as a pre-chaos check
|
||||
* **EoT:** Executed at the End of Test as a post-chaos check
|
||||
* **Edge:** Executed both, before and after the chaos
|
||||
* **Continuous:** The probe is executed continuously, with a specified polling interval during the chaos injection.
|
||||
* **OnChaos:** The probe is executed continuously, with a specified polling interval strictly for chaos duration of chaos
|
||||
|
||||
All probes share some common attributes:
|
||||
|
||||
* **probeTimeout:** Represents the time limit for the probe to execute the check specified and return the expected data.
|
||||
* **retry:** The number of times a check is re-run upon failure in the first attempt before declaring the probe status as failed.
|
||||
* **interval:** The period between subsequent retries
|
||||
* **probePollingInterval:** The time interval for which continuous probe should be sleep after each iteration
|
||||
* **initialDelaySeconds:** Represents the initial waiting time interval for the probes.
|
||||
|
||||
## Types of Litmus Probes
|
||||
|
||||
### **httpProbe**
|
||||
|
||||
The `httpProbe` allows developers to specify a URL which the experiment uses to gauge health/service availability (or other custom conditions) as part of the entry/exit criteria. The received status code is mapped against an expected status. It supports http `Get` and `Post` methods.
|
||||
|
||||
In HTTP `Get` method it sends a http `GET` request to the provided url and matches the response code based on the given criteria(`==`, `!=`, `oneOf`).
|
||||
|
||||
In HTTP `Post` method it sends a http `POST` request to the provided url. The http body can be provided in the `body` field. In the case of a complex POST request in which the body spans multiple lines, the `bodyPath` attribute can be used to provide the path to a file consisting of the same. This file can be made available to the experiment pod via a ConfigMap resource, with the ConfigMap name being defined in the ChaosEngine OR the ChaosExperiment CR.
|
||||
It can be defined at `.spec.experiments[].spec.probe` inside ChaosEngine.
|
||||
|
||||
> **NOTE:** `body` and `bodyPath` are mutually exclusive.
|
||||
|
||||
```yaml
|
||||
probe:
|
||||
- name: "check-frontend-access-url"
|
||||
type: "httpProbe"
|
||||
httpProbe/inputs:
|
||||
url: "<url>"
|
||||
insecureSkipVerify: false
|
||||
method:
|
||||
get:
|
||||
criteria: == # supports == & != and oneof operations
|
||||
responseCode: "<response code>"
|
||||
mode: "Continuous"
|
||||
runProperties:
|
||||
probeTimeout: 5
|
||||
interval: 5
|
||||
retry: 1
|
||||
probePollingInterval: 2
|
||||
```
|
||||
|
||||
The `httpProbe` is better used in the Continuous mode of operation as a parallel liveness indicator of a target or downstream application. It uses the `probePollingInterval` property to specify the polling interval for the access checks.
|
||||
|
||||
> **NOTE:** `insecureSkipVerify` can be set to true to skip the certificate checks.
|
||||
|
||||
<br />
|
||||
|
||||
### **cmdProbe**
|
||||
|
||||
The `cmdProbe` allows developers to run shell commands and match the resulting output as part of the entry/exit criteria. The intent behind this probe was to allow users to implement a non-standard & imperative way for expressing their hypothesis. For example, the cmdProbe enables you to check for specific data within a database, parse the value out of a JSON blob being dumped into a certain path or check for the existence of a particular string in the service logs.
|
||||
|
||||
In order to enable this behaviour, the probe supports an inline mode in which the command is run from within the experiment image as well as a source mode, where the command execution is carried out from within a new pod whose image can be specified. While inline is preferred for simple shell commands , source mode can be used when application-specific binaries are required. The `cmdProbe` can be defined at `.spec.experiments[].spec.probe` the path inside the ChaosEngine.
|
||||
|
||||
```yaml
|
||||
probe:
|
||||
- name: "check-database-integrity"
|
||||
type: "cmdProbe"
|
||||
cmdProbe/inputs:
|
||||
command: "<command>"
|
||||
comparator:
|
||||
type: "string" # supports: string, int, float
|
||||
criteria: "contains" #supports >=,<=,>,<,==,!= for int and contains,equal,notEqual,matches,notMatches for string values
|
||||
value: "<value-for-criteria-match>"
|
||||
source: "<repo>/<tag>" # it can be “inline” or any image
|
||||
mode: "Edge"
|
||||
runProperties:
|
||||
probeTimeout: 5
|
||||
interval: 5
|
||||
retry: 1
|
||||
initialDelaySeconds: 5
|
||||
```
|
||||
|
||||
<br />
|
||||
|
||||
### **k8sProbe**
|
||||
|
||||
With the proliferation of custom resources & operators, especially in the case of stateful applications, the steady-state is manifested as status parameters/flags within Kubernetes resources. k8sProbe addresses verification of the desired resource state by allowing users to define the Kubernetes GVR (group-version-resource) with appropriate filters (field selectors/label selectors). The experiment makes use of the Kubernetes Dynamic Client to achieve this. The `k8sProbe` can be defined at `.spec.experiments[].spec.probe` the path inside ChaosEngine.
|
||||
|
||||
It supports following CRUD operations which can be defined at `probe.operation`.
|
||||
|
||||
* **create:** It creates kubernetes resource based on the data provided inside probe.data field.
|
||||
* **delete:** It deletes matching kubernetes resource via GVR and filters (field selectors/label selectors).
|
||||
* **present:** It checks for the presence of kubernetes resource based on GVR and filters (field selectors/labelselectors).
|
||||
* **absent:** It checks for the absence of kubernetes resource based on GVR and filters (field selectors/labelselectors).
|
||||
|
||||
```yaml
|
||||
probe:
|
||||
- name: "check-app-cluster-cr-status"
|
||||
type: "k8sProbe"
|
||||
k8sProbe/inputs:
|
||||
command:
|
||||
group: "<appGroup>"
|
||||
version: "<appVersion>"
|
||||
resource: "<appResource>"
|
||||
namespace: "default"
|
||||
fieldSelector: "metadata.name=<appResourceName>,status.phase=Running"
|
||||
labelSelector: "<app-labels>"
|
||||
operation: "present" # it can be present, absent, create, delete
|
||||
mode: "EOT"
|
||||
runProperties:
|
||||
probeTimeout: 5
|
||||
interval: 5
|
||||
retry: 1
|
||||
```
|
||||
|
||||
<br />
|
||||
|
||||
### **promProbe**
|
||||
|
||||
The `promProbe` allows users to run Prometheus queries and match the resulting output against specific conditions. The intent behind this probe is to allow users to define metrics-based SLOs in a declarative way and determine the experiment verdict based on its success. The probe runs the query on a Prometheus server defined by the `endpoint`, and checks whether the output satisfies the specified `criteria`.
|
||||
|
||||
The promql query can be provided in the `query` field. In the case of complex queries that span multiple lines, the `queryPath` attribute can be used to provide the link to a file consisting of the query. This file can be made available in the experiment pod via a ConfigMap resource, with the ConfigMap being passed in the ChaosEngine OR the ChaosExperiment CR.
|
||||
|
||||
> **NOTE:** `query` and `queryPath` are mutually exclusive.
|
||||
|
||||
```yaml
|
||||
probe:
|
||||
- name: "check-probe-success"
|
||||
type: "promProbe"
|
||||
promProbe/inputs:
|
||||
endpoint: "<prometheus-endpoint>"
|
||||
query: "<promql-query>"
|
||||
comparator:
|
||||
criteria: "==" #supports >=,<=,>,<,==,!= comparision
|
||||
value: "<value-for-criteria-match>"
|
||||
mode: "Edge"
|
||||
runProperties:
|
||||
probeTimeout: 5
|
||||
interval: 5
|
||||
retry: 1
|
||||
```
|
||||
|
||||
<br />
|
||||
|
||||
---
|
||||
|
||||
## **Probe Status & Deriving Inferences**
|
||||
|
||||
The litmus chaos experiments run the probes defined in the ChaosEngine and update their stage-wise success in the ChaosResult custom resource, with details including the overall `probeSuccessPercentage` (a ratio of successful checks v/s total probes) and failure step, where applicable. The success of a probe is dependent on whether the expected status/results are met and also on whether it is successful in all the experiment phases defined by the probe’s execution mode. For example, probes that are executed in “Edge” mode, need the checks to be successful both during the pre-chaos & post-chaos phases to be declared as successful.
|
||||
|
||||
The pass criteria for an experiment is the logical conjunction of all probes defined in the ChaosEngine and an inbuilt entry/exit criteria. Failure of either indicates a failed hypothesis and is deemed experiment failure.
|
||||
|
||||
Provided below is a ChaosResult snippet containing the probe status for a mixed-probe ChaosEngine.
|
||||
|
||||
```yaml
|
||||
Name: app-pod-delete
|
||||
Namespace: test
|
||||
Labels: name=app-pod-delete
|
||||
Annotations: <none>
|
||||
API Version: litmuschaos.io/v1alpha1
|
||||
Kind: ChaosResult
|
||||
Metadata:
|
||||
Creation Timestamp: 2020-08-29T08:28:26Z
|
||||
Generation: 36
|
||||
Resource Version: 50239
|
||||
Self Link: /apis/litmuschaos.io/v1alpha1/namespaces/test/ChaosResults/app-pod-delete
|
||||
UID: b9e3638a-b7a4-4b93-bfea-bd143d91a5e8
|
||||
Spec:
|
||||
Engine: probe
|
||||
Experiment: pod-delete
|
||||
Status:
|
||||
Experimentstatus:
|
||||
Fail Step: N/A
|
||||
Phase: Completed
|
||||
Probe Success Percentage: 100
|
||||
Verdict: Pass
|
||||
|
||||
Probe Status:
|
||||
Name: check-frontend-access-url
|
||||
Status:
|
||||
Continuous: Passed 👍
|
||||
Type: HTTPProbe
|
||||
|
||||
Name: check-app-cluster-cr-status
|
||||
Status:
|
||||
Post Chaos: Passed 👍 #EoT
|
||||
Type: K8sProbe
|
||||
|
||||
Name: check-database-integrity
|
||||
Status:
|
||||
Post Chaos: Passed 👍 #Edge
|
||||
Pre Chaos: Passed 👍
|
||||
Type: CmdProbe
|
||||
Events:
|
||||
Type Reason Age From Message
|
||||
---- ------ ---- ---- -------
|
||||
Normal Summary 7s pod-delete-0s2jt6-s4rdx pod-delete experiment has been Passed
|
||||
```
|
||||
<br />
|
||||
|
||||
---
|
||||
|
||||
## **Probe Chaining**
|
||||
|
||||
Probe chaining enables reuse of probe a result (represented by the template function `{{ .<probeName>.probeArtifact.Register}})` in subsequent "downstream" probes defined in the ChaosEngine. Note that the order of execution of probes in the experiment depends purely on the order in which they are defined in the ChaosEngine.
|
||||
|
||||
Probe chaining is currently supported only for `cmdProbes`.
|
||||
|
||||
```yaml
|
||||
probe:
|
||||
- name: "probe1"
|
||||
type: "cmdProbe"
|
||||
cmdProbe/inputs:
|
||||
command: "<command>"
|
||||
comparator:
|
||||
type: "string"
|
||||
criteria: "equals"
|
||||
value: "<value-for-criteria-match>"
|
||||
source: "inline"
|
||||
mode: "SOT"
|
||||
runProperties:
|
||||
probeTimeout: 5
|
||||
interval: 5
|
||||
retry: 1
|
||||
- name: "probe2"
|
||||
type: "cmdProbe"
|
||||
cmdProbe/inputs:
|
||||
## probe1's result being used as one of the args in probe2
|
||||
command: "<commmand> {{ .probe1.ProbeArtifacts.Register }} <arg2>"
|
||||
comparator:
|
||||
type: "string"
|
||||
criteria: "equals"
|
||||
value: "<value-for-criteria-match>"
|
||||
source: "inline"
|
||||
mode: "SOT"
|
||||
runProperties:
|
||||
probeTimeout: 5
|
||||
interval: 5
|
||||
retry: 1
|
||||
```
|
||||
|
|
@ -0,0 +1,22 @@
|
|||
---
|
||||
id: schedule
|
||||
title: How to Schedule a Workflow
|
||||
sidebar_label: Schedule
|
||||
---
|
||||
|
||||
## Workflow Scheduling
|
||||
|
||||
---
|
||||
|
||||
### 1. Scheduling
|
||||
|
||||
After selecting the reliability scores for the tests, the user can now schedule the workflows. The default option in this screen is **Schedule Now**. With this option the user can run a workflow immediately once the workflow creation is completed. The second option available to the user is **Schedule Later**. With this option, the user can schedule the workflow at any time in the future. When this option is selected , the kind of the workflow changes from **Workflow** to **CronWorkflow** with a Cron syntax added in the workflow manifest. The following options are available to the user for scheduling a workflow later:
|
||||
|
||||
- Every Hour
|
||||
- Every Day
|
||||
- Every Week
|
||||
- Every Month
|
||||
|
||||
### 2. Verify and Commit
|
||||
|
||||
After adding the schedules, the user can see the verify and commit screen where all the details of the workflow are present such as the workflow name, the target cluster, workflow manifest, test weights etc. Once the user finishes configuring the workflow, it can be started by selecting the **Finish** button.
|
||||
|
|
@ -0,0 +1,65 @@
|
|||
---
|
||||
id: settings
|
||||
title: Settings
|
||||
sidebar_label: Settings
|
||||
---
|
||||
|
||||
---
|
||||
Settings section allows a user to manage user related information and settings. At highlevel the settings section can be classified into the following
|
||||
|
||||
1. My Account
|
||||
2. Team
|
||||
3. User Management
|
||||
4. GitOps
|
||||
|
||||
|
||||
## My Account
|
||||
|
||||
Here you can edit your personal details such as full Name, email-id, and password. Username can’t be changed and to update the password you are also required to enter your current password. In case you forget your current password, you need to contact the Admin for a password reset which you can change later in this section.
|
||||
|
||||
**Note:** Edit Avatar feature will be available in the upcoming releases.
|
||||
|
||||
## Team
|
||||
|
||||
Teaming is based on the following principles and each user can have one of the 3 user roles:
|
||||
1. Owner
|
||||
2. Viewer
|
||||
3. Editor
|
||||
- Owner: A user would be the owner of his/her project by default. Only one user can be owner of one project
|
||||
|
||||
- Owner can invite other users using the following roles:
|
||||
- Viewer: Can view the project, like view the workflow schedule, but can’t create or delete any workflow schedules
|
||||
- Editor: Can make changes to the project, like view, edit, delete and create the workflow schedules
|
||||
|
||||
- Settings page will not be visible to the user if he/she is browsing someone else’s project. The user can view the settings page if the user is the project owner.
|
||||
|
||||
- On the teaming tab, owner can view the list of team members with other details including their role in the project, email-id, date-time of joining the team of the project. Owner can view/send/accept an invitation. Owner can remove a member from his/her project. User can also exit a project of which he/she is a part of.
|
||||
|
||||
## User Management
|
||||
|
||||
User Management section is only available to Admin users, where the Admin can:
|
||||
|
||||
### 1. Create User
|
||||
|
||||
Create a new user by providing details such as full name, the username(mandatory), email, and password(mandatory). Please remember that the username can’t be changed later.
|
||||
|
||||
**Note:** You might get an error while creating a user that `user already exists. This can occur if the username is already taken by anyone else. In that case, you can try changing the username and create again.
|
||||
|
||||
### 2. View user list
|
||||
|
||||
This section displays a list of users created by the Admin along with other details such as their current status(currently logged in or not), full name, username, email-id, date and time of the user creation, and a menu containing an option for password reset:
|
||||
|
||||
- **Edit profile:** Here, the Admin can reset the password of other users.
|
||||
|
||||
## GitOps
|
||||
|
||||
This section lets you choose where to store the workflow configuration. You can either store it locally in Litmus or in a Git repository. If you want to store it in litmus, you can just choose the first option(by default the first option is selected). Else if you want to store it in a git repository, you can do the following steps:
|
||||
|
||||
1. You need to have an access to a repository or create an empty repository, then enter the Git URL and branch name in the given fields in which you want to push the workflow configuration.
|
||||
2. Then you can either choose to provide by either of the following methods
|
||||
- **Access token**- User needs to provide the access token here, which can be generated from your Git provider by going to `Developer Settings` → `Personal Access token` → `Generate New token`. Make sure you have write access.
|
||||
- **SSH** - User needs to copy the generated key and add it to your repository by going to `Settings(of the repo)` →`Deploy Keys` → `Add deploy keys`. Make sure you check the `Allow write access` checkbox.
|
||||
3. Once the Step 2 is done, click on the `connect` button to link it to your repository. Now every time you run a workflow, the configurations will get pushed to your repository.
|
||||
|
||||
***Note*** On selecting GitOps, only new workflows will be synced to git repository. Any existing or active workflows saved locally will not be synced into the git repository.
|
||||
|
||||
|
|
@ -0,0 +1,30 @@
|
|||
---
|
||||
id: update
|
||||
title: How to Update a Workflow
|
||||
sidebar_label: Update
|
||||
---
|
||||
|
||||
## Update a Workflow
|
||||
|
||||
---
|
||||
|
||||
After scheduling a workflow, the user can update the scheduled configuration of the workflow by clicking on Schedules tab.
|
||||
Here the user can view several options like:
|
||||
- Re-Run schedule
|
||||
- Edit Schedule
|
||||
- Delete Schedule
|
||||
- Download Manifest
|
||||
|
||||
### 1. Re-Run Schedule
|
||||
This option allows the user to re-run a workflow with the same configuration as earlier. This option is only available when the workflow is scheduled once.
|
||||
|
||||
### 2. Edit Schedule
|
||||
This option allows the user to edit the schedule configuration of the workflow. The user can either disable the workflow or change the Cron Syntax by changing the scheduled timings.
|
||||
The disable workflow option adds a label **suspend: true** that disables any scheduled workflow.
|
||||
This option is only available if the workflow is a Cron Workflow i.e a recurring workflow.
|
||||
|
||||
### 3. Delete Schedule
|
||||
This option allows the user to delete a scheduled workflow. This is an irreversible process and once the delete option is selected, all the workflow configuration and the schedules are deleted from the portal.
|
||||
|
||||
### 4. Download Manifest
|
||||
This option allows the user to download the manifest of the workflow. The manifest consists of all the configurations that the user has added while workflow creation. The user can then alter changes in this manifest and upload it with the portal to schedule a new workflow.
|
||||
|
|
@ -0,0 +1,151 @@
|
|||
{
|
||||
"version-2.0.0-Beta-8/docs": [
|
||||
{
|
||||
"collapsed": true,
|
||||
"type": "category",
|
||||
"label": "Introduction",
|
||||
"items": [
|
||||
{
|
||||
"type": "doc",
|
||||
"id": "version-2.0.0-Beta-8/introduction"
|
||||
},
|
||||
{
|
||||
"type": "doc",
|
||||
"id": "version-2.0.0-Beta-8/architecture"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"collapsed": true,
|
||||
"type": "category",
|
||||
"label": "Getting Started",
|
||||
"items": [
|
||||
{
|
||||
"type": "doc",
|
||||
"id": "version-2.0.0-Beta-8/getstarted"
|
||||
},
|
||||
{
|
||||
"collapsed": true,
|
||||
"type": "category",
|
||||
"label": "Installation",
|
||||
"items": [
|
||||
{
|
||||
"type": "doc",
|
||||
"id": "version-2.0.0-Beta-8/litmus-install-cluster-mode"
|
||||
},
|
||||
{
|
||||
"type": "doc",
|
||||
"id": "version-2.0.0-Beta-8/litmus-install-namespace-mode"
|
||||
},
|
||||
{
|
||||
"type": "doc",
|
||||
"id": "version-2.0.0-Beta-8/agent-install"
|
||||
},
|
||||
{
|
||||
"type": "doc",
|
||||
"id": "version-2.0.0-Beta-8/litmus-with-ingress"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "doc",
|
||||
"id": "version-2.0.0-Beta-8/create-workflow"
|
||||
},
|
||||
{
|
||||
"type": "doc",
|
||||
"id": "version-2.0.0-Beta-8/observe-workflow"
|
||||
},
|
||||
{
|
||||
"collapsed": true,
|
||||
"type": "category",
|
||||
"label": "Uninstallation",
|
||||
"items": [
|
||||
{
|
||||
"type": "doc",
|
||||
"id": "version-2.0.0-Beta-8/agent-uninstall"
|
||||
},
|
||||
{
|
||||
"type": "doc",
|
||||
"id": "version-2.0.0-Beta-8/litmus-uninstall"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"collapsed": true,
|
||||
"type": "category",
|
||||
"label": "User Guide",
|
||||
"items": [
|
||||
{
|
||||
"collapsed": true,
|
||||
"type": "category",
|
||||
"label": "Day 0",
|
||||
"items": [
|
||||
{
|
||||
"type": "doc",
|
||||
"id": "version-2.0.0-Beta-8/d0-create"
|
||||
},
|
||||
{
|
||||
"type": "doc",
|
||||
"id": "version-2.0.0-Beta-8/d0-observe"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"collapsed": true,
|
||||
"type": "category",
|
||||
"label": "Day 1",
|
||||
"items": [
|
||||
{
|
||||
"type": "doc",
|
||||
"id": "version-2.0.0-Beta-8/d1-schedule"
|
||||
},
|
||||
{
|
||||
"type": "doc",
|
||||
"id": "version-2.0.0-Beta-8/d1-update"
|
||||
},
|
||||
{
|
||||
"type": "doc",
|
||||
"id": "version-2.0.0-Beta-8/settings"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"collapsed": true,
|
||||
"type": "category",
|
||||
"label": "Day 2",
|
||||
"items": [
|
||||
{
|
||||
"type": "doc",
|
||||
"id": "version-2.0.0-Beta-8/myhub"
|
||||
},
|
||||
{
|
||||
"type": "doc",
|
||||
"id": "version-2.0.0-Beta-8/gitops"
|
||||
},
|
||||
{
|
||||
"type": "doc",
|
||||
"id": "version-2.0.0-Beta-8/probes"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"collapsed": true,
|
||||
"type": "category",
|
||||
"label": "Advanced",
|
||||
"items": [
|
||||
{
|
||||
"type": "doc",
|
||||
"id": "version-2.0.0-Beta-8/litmus-psp"
|
||||
},
|
||||
{
|
||||
"type": "doc",
|
||||
"id": "version-2.0.0-Beta-8/k8s-support"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -1 +1,4 @@
|
|||
["2.0.0-Beta-3"]
|
||||
[
|
||||
"2.0.0-Beta-8",
|
||||
"2.0.0-Beta-3"
|
||||
]
|
||||
|
|
|
|||
Loading…
Reference in New Issue