Added docs for 2.12.0 and updated litmusctl docs (#200)
* Added docs for 2.12.0 and updated litmusctl docs Signed-off-by: Sarthak Jain <sarthak.jain@harness.io> * Added changes for 2.12.0 links updates Signed-off-by: Vedant <vedant.srotria@harness.io> Signed-off-by: Sarthak Jain <sarthak.jain@harness.io> Signed-off-by: Vedant <vedant.srotria@harness.io> Co-authored-by: Vedant <vedant.srotria@harness.io>
This commit is contained in:
Sarthak Jain
GitHub
parent
e2d3824507
commit
1df690b390
|
|
@ -85,3 +85,11 @@ Once GitOps is enabled, any new chaos scenarios created will be stored in the co
|
|||
### How to solve `invalid token` issue in litmusctl?
|
||||
|
||||
You can use the command `litmusctl config set-account` to generate a new token. You get `invalid token` error if the authorization token is expired. Authorization token has a life span of 24 hours.
|
||||
|
||||
### How to check if installed version of litmus control plane is compatible with installed litmusctl?
|
||||
|
||||
litmusctl will prompt if your installed litmusctl and litmus control plane are compatible when user runs `litmusctl config set-account`
|
||||
|
||||
### How to check compatibility of litmusctl with litmus control plane?
|
||||
|
||||
You can use command `litmusctl version` to check the compatibility of litmusctl or you can refer to: https://github.com/litmuschaos/litmusctl#compatibility-matrix to get the compatibility matrix of litmusctl and litmus control plane.
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
---
|
||||
id: chaos-workflow-creation
|
||||
title: Create Scenarios using Litmusctl
|
||||
sidebar_label: Create Scenarios
|
||||
sidebar_label: Create Chaos Scenarios
|
||||
---
|
||||
|
||||
---
|
||||
|
|
@ -44,26 +44,26 @@ litmusctl config set-account --endpoint="" --username="" --password=""
|
|||
> Note:
|
||||
>
|
||||
> - To get `project-id`, apply `litmusctl get projects`
|
||||
> - To get `agent-id`, apply `litmusctl get agents --project-id=""`
|
||||
> - To get `chaos-delegate-id`, apply `litmusctl get chaos-delegates --project-id=""`
|
||||
|
||||
```shell
|
||||
litmusctl create workflow -f custom-chaos-workflow.yml --project-id="" --agent-id=""
|
||||
litmusctl create chaos-scenario -f custom-chaos-scenario.yml --project-id="" --chaos-delegate-id=""
|
||||
```
|
||||
|
||||
#### Verify the new Chaos Scenario
|
||||
|
||||
To verify the successful creation, you can either view the list of chaos Scenarios at the ChaosCenter dashboard or run the below given command to list the chaos workflow within a given project.
|
||||
To verify the successful creation, you can either view the list of chaos scenarios at the ChaosCenter dashboard or run the below given command to list the chaos scenarios within a given project.
|
||||
|
||||
```shell
|
||||
litmusctl get workflows --project-id=""
|
||||
litmusctl get chaos-scenarios --project-id=""
|
||||
```
|
||||
|
||||
**Output:**
|
||||
|
||||
```
|
||||
WORKFLOW ID WORKFLOW NAME WORKFLOW TYPE NEXT SCHEDULE AGENT ID AGENT NAME LAST UPDATED BY
|
||||
9433b48c-4ab7-4544-8dab-4a7237619e09 custom-chaos-workflow-1627980541 Non Cron Workflow None f9799723-29f1-454c-b830-ae8ba7ee4c30 Self-Agent admin
|
||||
Showing 1 of 1 workflows
|
||||
CHAOS SCENARIO ID CHAOS SCENARIO NAME CHAOS SCENARIO TYPE NEXT SCHEDULE CHAOS DELEGATE ID CHAOS DELEGATE NAME LAST UPDATED BY
|
||||
9433b48c-4ab7-4544-8dab-4a7237619e09 custom-chaos-scenario-1627980541 Non Cron Scenario None f9799723-29f1-454c-b830-ae8ba7ee4c30 Self-Chaos-Delegate admin
|
||||
Showing 1 of 1 chaos scenarios
|
||||
```
|
||||
|
||||
---
|
||||
|
|
@ -73,21 +73,21 @@ Showing 1 of 1 workflows
|
|||
- To list all the chaos scenario runs within a project, issue the following command.
|
||||
|
||||
```shell
|
||||
litmusctl get workflowruns --project-id=""
|
||||
litmusctl get chaos-scenario-runs --project-id=""
|
||||
```
|
||||
|
||||
**Output:**
|
||||
|
||||
```
|
||||
WORKFLOW RUN ID STATUS RESILIENCY SCORE WORKFLOW ID WORKFLOW NAME TARGET AGENT LAST RUN EXECUTED BY
|
||||
8ceb712c-1ed4-40e6-adc4-01f78d281506 Running 0.00 9433b48c-4ab7-4544-8dab-4a7237619e09 custom-chaos-workflow-1627980541 Self-Agent June 1 2022, 10:28:02 pm admin
|
||||
Showing 1 of 1 workflow runs
|
||||
CHAOS SCENARIO RUN ID STATUS RESILIENCY SCORE CHAOS SCENARIO ID CHAOS SCENARIO NAME TARGET CHAOS DELEGATE LAST RUN EXECUTED BY
|
||||
8ceb712c-1ed4-40e6-adc4-01f78d281506 Running 0.00 9433b48c-4ab7-4544-8dab-4a7237619e09 custom-chaos-scenario-1627980541 Self-Chaos-Delegate June 1 2022, 10:28:02 pm admin
|
||||
Showing 1 of 1 scenario runs
|
||||
```
|
||||
|
||||
- To describe a particular chaos scenario, issue the following command.
|
||||
|
||||
```shell
|
||||
litmusctl describe workflow 9433b48c-4ab7-4544-8dab-4a7237619e09 --project-id=""
|
||||
litmusctl describe chaos-scenario 9433b48c-4ab7-4544-8dab-4a7237619e09 --project-id=""
|
||||
```
|
||||
|
||||
**Output:**
|
||||
|
|
@ -99,10 +99,10 @@ metadata:
|
|||
creationTimestamp: null
|
||||
labels:
|
||||
cluster_id: f9799723-29f1-454c-b830-ae8ba7ee4c30
|
||||
subject: custom-chaos-workflow_litmus
|
||||
subject: custom-chaos-scenario_litmus
|
||||
workflow_id: 9433b48c-4ab7-4544-8dab-4a7237619e09
|
||||
workflows.argoproj.io/controller-instanceid: f9799723-29f1-454c-b830-ae8ba7ee4c30
|
||||
name: custom-chaos-workflow-1627980541
|
||||
name: custom-chaos-scenario-1627980541
|
||||
namespace: litmus
|
||||
spec:
|
||||
...
|
||||
|
|
@ -111,13 +111,13 @@ spec:
|
|||
- To delete a particular chaos scenario, issue the following command.
|
||||
|
||||
```shell
|
||||
litmusctl delete workflow df91c6b2-ad33-45ae-9a2f-00cb87978657 --project-id=""
|
||||
litmusctl delete chaos-scenario df91c6b2-ad33-45ae-9a2f-00cb87978657 --project-id=""
|
||||
```
|
||||
|
||||
**Output:**
|
||||
|
||||
```
|
||||
🚀 ChaosWorkflow successfully deleted.
|
||||
🚀 Chaos scenario successfully deleted.
|
||||
```
|
||||
|
||||
For more information related to flags, Use `litmusctl --help`.
|
||||
|
|
|
|||
|
|
@ -6,7 +6,7 @@ sidebar_label: Installation
|
|||
|
||||
---
|
||||
|
||||
The Litmuschaos command-line tool, litmusctl, allows you to manage litmuschaos's chaos delegate plane. You can use litmusctl to create chaos delegates, project, and manage multiple litmuschaos accounts.
|
||||
The Litmuschaos command-line tool, litmusctl, allows you to manage litmuschaos's chaos delegate plane. You can use litmusctl to connect and disconnect chaos delegates, create chaos scenarios, project, and manage multiple litmuschaos accounts.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
|
|
@ -20,91 +20,140 @@ Litmusctl CLI requires the following things:
|
|||
|
||||
For more information including a complete list of litmusctl operations, see the litmusctl reference documentation.
|
||||
|
||||
- For v0.3.0 or latest: <a href="https://github.com/litmuschaos/litmusctl/blob/master/Usage.md">Click here</a>
|
||||
- For v0.12.0 or latest: <a href="https://github.com/litmuschaos/litmusctl/blob/master/Usage.md">Click here</a>
|
||||
- For v0.2.0 or earlier: <a href="https://github.com/litmuschaos/litmusctl/blob/master/Usage_v0.2.0.md">Click here</a>
|
||||
|
||||
|
||||
## Compatibility matrix
|
||||
|
||||
To check compatibility of litmusctl with Chaos Center
|
||||
|
||||
<table>
|
||||
<th>litmusctl version</th>
|
||||
<th>Chaos Center supported versions</th>
|
||||
|
||||
<tr>
|
||||
<td>0.6.0</td>
|
||||
<td>2.2.0, 2.3.0</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>0.7.0</td>
|
||||
<td>2.4.0, 2.5.0, 2.6.0, 2.7.0, 2.8.0</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>0.8.0</td>
|
||||
<td>2.4.0, 2.5.0, 2.6.0, 2.7.0, 2.8.0</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>0.9.0</td>
|
||||
<td>2.4.0, 2.5.0, 2.6.0, 2.7.0, 2.8.0</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>0.10.0</td>
|
||||
<td>2.9.0, 2.10.0, 2.11.0, 2.12.0</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>0.11.0</td>
|
||||
<td>2.9.0, 2.10.0, 2.11.0, 2.12.0</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>0.12.0</td>
|
||||
<td>2.9.0, 2.10.0, 2.11.0, 2.12.0</td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
## Installation
|
||||
|
||||
To install the latest version of litmusctl follow the below steps:
|
||||
|
||||
<table>
|
||||
<th>Platforms</th>
|
||||
<th>v0.12.0</th>
|
||||
<th>v0.11.0</th>
|
||||
<th>v0.10.0</th>
|
||||
<th>v0.9.0</th>
|
||||
<th>v0.7.0</th>
|
||||
<th>v0.6.0</th>
|
||||
<th>v0.5.0</th>
|
||||
<th>master(Unreleased)</th>
|
||||
<tr>
|
||||
<td>litmusctl-darwin-amd64 (MacOS)</td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-darwin-amd64-0.12.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-darwin-amd64-0.11.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-darwin-amd64-0.10.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-darwin-amd64-v0.9.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-darwin-amd64-v0.7.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-darwin-amd64-v0.6.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-darwin-amd64-v0.5.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-darwin-amd64-master.tar.gz">Click here</a></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>litmusctl-linux-386</td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-linux-386-0.12.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-linux-386-0.11.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-linux-386-0.10.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-linux-386-v0.9.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-linux-386-v0.7.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-linux-386-v0.6.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-linux-386-v0.5.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-linux-386-master.tar.gz">Click here</a></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>litmusctl-linux-amd64</td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-linux-amd64-0.12.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-linux-amd64-0.11.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-linux-amd64-0.10.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-linux-amd64-v0.9.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-linux-amd64-v0.7.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-linux-amd64-v0.6.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-linux-amd64-v0.5.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-linux-amd64-master.tar.gz">Click here</a></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>litmusctl-linux-arm</td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-linux-arm-0.12.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-linux-arm-0.11.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-linux-arm-0.10.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-linux-arm-v0.9.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-linux-arm-v0.7.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-linux-arm-v0.6.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-linux-arm-v0.5.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-linux-arm-master.tar.gz">Click here</a></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>litmusctl-linux-arm64</td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-linux-arm64-0.12.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-linux-arm64-0.11.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-linux-arm64-0.10.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-linux-arm64-v0.9.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-linux-arm64-v0.7.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-linux-arm64-v0.6.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-linux-arm64-v0.5.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-linux-arm64-master.tar.gz">Click here</a></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>litmusctl-windows-386</td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-windows-386-0.12.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-windows-386-0.11.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-windows-386-0.10.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-windows-386-v0.9.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-windows-386-v0.7.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-windows-386-v0.6.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-windows-386-v0.5.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-windows-386-master.tar.gz">Click here</a></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>litmusctl-windows-amd64</td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-windows-amd64-0.12.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-windows-amd64-0.11.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-windows-amd64-0.10.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-windows-amd64-v0.9.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-windows-amd64-v0.7.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-windows-amd64-v0.6.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-windows-amd64-v0.5.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-windows-amd64-master.tar.gz">Click here</a></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>litmusctl-windows-arm</td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-windows-arm-0.12.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-windows-arm-0.11.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-windows-arm-0.10.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-windows-arm-0.10.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-windows-arm-v0.9.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-windows-arm-v0.7.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-windows-arm-v0.6.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-windows-arm-v0.5.0.tar.gz">Click here</a></td>
|
||||
<td><a href="https://litmusctl-production-bucket.s3.amazonaws.com/litmusctl-windows-arm-master.tar.gz">Click here</a></td>
|
||||
</tr>
|
||||
</table>
|
||||
|
|
|
|||
|
|
@ -6,12 +6,12 @@ sidebar_label: Using interactive mode
|
|||
|
||||
---
|
||||
|
||||
# Usage: Litmusctl v0.10.0
|
||||
# Usage: Litmusctl v0.12.0 (Interactive mode)
|
||||
|
||||
> Notes:
|
||||
>
|
||||
> - For litmusctl v0.10.0 or latest
|
||||
> - Compatible with Litmus 2.9.0 or latest
|
||||
> - For litmusctl v0.12.0 or latest
|
||||
> - Compatible with Litmus 2.12.0 or latest
|
||||
|
||||
### litmusctl Syntax
|
||||
|
||||
|
|
@ -21,8 +21,8 @@ sidebar_label: Using interactive mode
|
|||
litmusctl [command] [TYPE] [flags]
|
||||
```
|
||||
|
||||
- Command: refers to what you do want to perform (create, get and config)
|
||||
- Type: refers to the feature type you are performing a command against (Chaos Delegate , project etc.)
|
||||
- Command: refers to what you do want to perform (connect, create, get and config)
|
||||
- Type: refers to the feature type you are performing a command against (chaos-delegate, project etc.)
|
||||
- Flags: It takes some additional information for resource operations. For example, `--installation-mode` allows you to specify an installation mode.
|
||||
|
||||
Litmusctl is using the `.litmusconfig` config file to manage multiple accounts
|
||||
|
|
@ -32,7 +32,7 @@ Litmusctl is using the `.litmusconfig` config file to manage multiple accounts
|
|||
|
||||
Litmusctl supports both interactive and non-interactive(flag based) modes.
|
||||
|
||||
> Only `litmusctl connect agent` command needs --non-interactive flag, other commands don't need this flag to be in non-interactive mode. If mandatory flags aren't passed, then litmusctl takes input in an interactive mode.
|
||||
> Only `litmusctl connect chaos-delegate` command needs --non-interactive flag, other commands don't need this flag to be in non-interactive mode. If mandatory flags aren't passed, then litmusctl takes input in an interactive mode.
|
||||
|
||||
Multiple external Chaos Delegate can be connected to the ChaosCenter with the help of the command line utility [litmusctl](installation.md)
|
||||
|
||||
|
|
@ -64,7 +64,7 @@ account.username/admin configured
|
|||
- To connect a Chaos Delegate in a cluster mode
|
||||
|
||||
```shell
|
||||
litmusctl connect agent
|
||||
litmusctl connect chaos-delegate
|
||||
```
|
||||
|
||||
There will be a list of existing projects displayed on the terminal. Select the desired project by entering the sequence number indicated against it.
|
||||
|
|
@ -96,11 +96,11 @@ Select Mode [Default: cluster] [Range: 1-2]: 1
|
|||
🏃 Running prerequisites check....
|
||||
🔑 clusterrole ✅
|
||||
🔑 clusterrolebinding ✅
|
||||
🌟 Sufficient permissions. Installing the Agent...
|
||||
🌟 Sufficient permissions. Installing the Chaos Delegate...
|
||||
|
||||
```
|
||||
|
||||
Next, enter the details of the new chaos delegate.
|
||||
Next, enter the details of the new Chaos Delegate.
|
||||
|
||||
Fields to be filled in <br />
|
||||
|
||||
|
|
@ -108,20 +108,24 @@ Fields to be filled in <br />
|
|||
<th>Field</th>
|
||||
<th>Description</th>
|
||||
<tr>
|
||||
<td>Agent Name:</td>
|
||||
<td>Enter a name of the agent which needs to be unique across the project</td>
|
||||
<td>Chaos Delegate Name:</td>
|
||||
<td>Enter a name of the Chaos Delegate which needs to be unique across the project</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Agent Description:</td>
|
||||
<td>Fill in details about the agent</td>
|
||||
<td>Chaos Delegate Description:</td>
|
||||
<td>Fill in details about the Chaos Delegate</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Skip SSL verification</td>
|
||||
<td>Choose whether Chaos Delegate will skip SSL/TLS verification</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Node Selector:</td>
|
||||
<td>To deploy the agent on a particular node based on the node selector labels</td>
|
||||
<td>To deploy the Chaos Delegate on a particular node based on the node selector labels</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Platform Name:</td>
|
||||
<td>Enter the platform name on which this agent is hosted. For example, AWS, GCP, Rancher etc.</td>
|
||||
<td>Enter the platform name on which this Chaos Delegate is hosted. For example, AWS, GCP, Rancher etc.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Enter the namespace:</td>
|
||||
|
|
@ -134,13 +138,15 @@ Fields to be filled in <br />
|
|||
</table>
|
||||
|
||||
```
|
||||
Enter the details of the agent
|
||||
Enter the details of the Chaos Delegate
|
||||
|
||||
Agent Name: New-Agent
|
||||
Chaos Delegate Name: New-Chaos-Delegate
|
||||
|
||||
Agent Description: This is a new agent
|
||||
Chaos Delegate Description: This is a new Chaos Delegate
|
||||
|
||||
Do you want NodeSelector to be added in the agent deployments (Y/N) (Default: N): N
|
||||
Do you want Chaos Delegate to skip SSL/TLS check (Y/N) (Default: N): n
|
||||
|
||||
Do you want NodeSelector to be added in the Chaos Delegate deployments (Y/N) (Default: N): N
|
||||
|
||||
Platform List:
|
||||
1. AWS
|
||||
|
|
@ -156,14 +162,15 @@ Enter the namespace (new or existing namespace) [Default: litmus]:
|
|||
```
|
||||
|
||||
Once, all these steps are implemented you will be able to see a summary of all the entered fields.
|
||||
After verification of these details, you can proceed with the connection of the chaos delegate by entering Y. The process of connection might take up to a few seconds.
|
||||
After verification of these details, you can proceed with the connection of the Chaos Delegate by entering Y. The process of connection might take up to a few seconds.
|
||||
|
||||
```
|
||||
Enter service account [Default: litmus]:
|
||||
|
||||
📌 Summary
|
||||
Agent Name: New-Agent
|
||||
Agent Description: This is a new agent
|
||||
Chaos Delegate Name: New-Chaos-Delegate
|
||||
Chaos Delegate Description: This is a new Chaos Delegate
|
||||
Chaos Delegate SSL/TLS Skip: false
|
||||
Platform Name: Others
|
||||
Namespace: litmus
|
||||
Service Account: litmus (new)
|
||||
|
|
@ -171,20 +178,37 @@ Service Account: litmus (new)
|
|||
Installation Mode: cluster
|
||||
|
||||
🤷 Do you want to continue with the above details? [Y/N]: Y
|
||||
👍 Continuing agent connection!!
|
||||
👍 Continuing Chaos Delegate connection!!
|
||||
Applying YAML:
|
||||
https://preview.litmuschaos.io/api/file/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbHVzdGVyX2lkIjoiMDUyZmFlN2UtZGM0MS00YmU4LWJiYTgtMmM4ZTYyNDFkN2I0In0.i31QQDG92X5nD6P_-7TfeAAarZqLvUTFfnAghJYXPiM.yaml
|
||||
|
||||
💡 Connecting agent to Litmus Portal.
|
||||
🏃 Agents are running!!
|
||||
💡 Connecting Chaos Delegate to ChaosCenter.
|
||||
🏃 Chaos Delegate is running!!
|
||||
|
||||
🚀 Agent Connection Successful!! 🎉
|
||||
👉 Litmus agents can be accessed here: https://preview.litmuschaos.io/targets
|
||||
🚀 Chaos Delegate Connection Successful!! 🎉
|
||||
👉 Litmus Chaos Delegates can be accessed here: https://preview.litmuschaos.io/targets
|
||||
```
|
||||
|
||||
#### Verify the new Chaos Delegate Connection\*\*
|
||||
|
||||
To verify, if the connection process was successful you can view the list of connected chaos delegates from the Targets section on your ChaosCenter and ensure that the connected chaos delegate is in Active State.
|
||||
To verify, if the connection process was successful you can view the list of connected Chaos Delegates from the Targets section on your ChaosCenter and ensure that the connected Chaos Delegate is in Active State.
|
||||
|
||||
---
|
||||
|
||||
### Steps to create a Chaos Scenario
|
||||
|
||||
* To setup an account with litmusctl
|
||||
```shell
|
||||
litmusctl config set-account --endpoint="" --username="" --password=""
|
||||
```
|
||||
|
||||
* To create a Chaos Scenario by passing a manifest file
|
||||
> Note:
|
||||
> * To get `project-id`, apply `litmusctl get projects`
|
||||
> * To get `chaos-delegate-id`, apply `litmusctl get chaos-delegates --project-id=""`
|
||||
```shell
|
||||
litmusctl create chaos-scenario -f custom-chaos-scenario.yml --project-id="" --chaos-delegate-id=""
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -260,10 +284,10 @@ PROJECT ID PROJECT NAME CREATEDAT
|
|||
7a4a259a-1ae5-4204-ae83-89a8838eaec3 DevOps Project 2021-07-21 14:39:14 +0530 IST
|
||||
```
|
||||
|
||||
- To get an overview of the chaos delegates available within a project, issue the following command.
|
||||
- To get an overview of the Chaos Delegates available within a project, issue the following command.
|
||||
|
||||
```shell
|
||||
litmusctl get agents
|
||||
litmusctl get chaos-delegates
|
||||
|
||||
Enter the Project ID: 50addd40-8767-448c-a91a-5071543a2d8e
|
||||
```
|
||||
|
|
@ -271,25 +295,125 @@ Enter the Project ID: 50addd40-8767-448c-a91a-5071543a2d8e
|
|||
**Output:**
|
||||
|
||||
```
|
||||
AGENTID AGENTNAME STATUS REGISTRATION
|
||||
55ecc7f2-2754-43aa-8e12-6903e4c6183a agent-1 ACTIVE REGISTERED
|
||||
13dsf3d1-5324-54af-4g23-5331g5v2364f agent-2 INACTIVE NOT REGISTERED
|
||||
CHAOS DELEGATE ID CHAOS DELEGATE NAME STATUS REGISTRATION
|
||||
55ecc7f2-2754-43aa-8e12-6903e4c6183a chaos-delegate-1 ACTIVE REGISTERED
|
||||
13dsf3d1-5324-54af-4g23-5331g5v2364f chaos-delegate-2 INACTIVE NOT REGISTERED
|
||||
```
|
||||
|
||||
- To disconnect an chaos delegate, issue the following command.
|
||||
|
||||
* To disconnect an Chaos Delegate, issue the following command..
|
||||
```shell
|
||||
litmusctl disconnect agent 55ecc7f2-2754-43aa-8e12-6903e4c6183a --project-id=""
|
||||
litmusctl disconnect chaos-delegate <chaos-delegate-id> --project-id=""
|
||||
```
|
||||
|
||||
**Output:**
|
||||
|
||||
```
|
||||
🚀 ChaosAgent successfully disconnected.
|
||||
🚀 Chaos Delegate successfully disconnected.
|
||||
```
|
||||
|
||||
For more information related to flags, Use `litmusctl --help`.
|
||||
|
||||
* To list the created Chaos Scenarios within a project, issue the following command.
|
||||
```shell
|
||||
litmusctl get chaos-scenarios --project-id=""
|
||||
```
|
||||
|
||||
**Output:**
|
||||
|
||||
```
|
||||
CHAOS SCENARIO ID CHAOS SCENARIO NAME CHAOS SCENARIO TYPE NEXT SCHEDULE CHAOS DELEGATE ID CHAOS DELEGATE NAME LAST UPDATED BY
|
||||
9433b48c-4ab7-4544-8dab-4a7237619e09 custom-chaos-scenario-1627980541 Non Cron Chaos Scenario None f9799723-29f1-454c-b830-ae8ba7ee4c30 Self-Chaos-delegate admin
|
||||
|
||||
Showing 1 of 1 Chaos Scenarios
|
||||
```
|
||||
|
||||
|
||||
* To list all the Chaos Scenario runs within a project, issue the following command.
|
||||
```shell
|
||||
litmusctl get chaos-scenario-runs --project-id=""
|
||||
```
|
||||
|
||||
**Output:**
|
||||
|
||||
```
|
||||
CHAOS SCENARIO RUN ID STATUS RESILIENCY SCORE CHAOS SCENARIO ID CHAOS SCENARIO NAME TARGET CHAOS DELEGATE LAST RUN EXECUTED BY
|
||||
8ceb712c-1ed4-40e6-adc4-01f78d281506 Running 0.00 9433b48c-4ab7-4544-8dab-4a7237619e09 custom-chaos-scenario-1627980541 Self-Chaos-Delegate June 1 2022, 10:28:02 pm admin
|
||||
|
||||
Showing 1 of 1 Chaos Scenario runs
|
||||
```
|
||||
|
||||
|
||||
* To describe a particular Chaos Scenario, issue the following command.
|
||||
```shell
|
||||
litmusctl describe chaos-scenario <chaos-scenario-id> --project-id=""
|
||||
```
|
||||
|
||||
**Output:**
|
||||
|
||||
```
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: Workflow
|
||||
metadata:
|
||||
creationTimestamp: null
|
||||
labels:
|
||||
cluster_id: f9799723-29f1-454c-b830-ae8ba7ee4c30
|
||||
subject: custom-chaos-scenario_litmus
|
||||
workflow_id: 9433b48c-4ab7-4544-8dab-4a7237619e09
|
||||
workflows.argoproj.io/controller-instanceid: f9799723-29f1-454c-b830-ae8ba7ee4c30
|
||||
name: custom-chaos-scenario-1627980541
|
||||
namespace: litmus
|
||||
spec:
|
||||
...
|
||||
```
|
||||
|
||||
|
||||
* To delete a particular Chaos Scenario, issue the following command.
|
||||
```shell
|
||||
litmusctl delete chaos-scenario <chaos-scenario-id> --project-id=""
|
||||
```
|
||||
|
||||
**Output:**
|
||||
|
||||
```
|
||||
🚀 Chaos Scenario successfully deleted.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Flag details
|
||||
|
||||
<table>
|
||||
<th>Flag</th>
|
||||
<th>Short Flag</th>
|
||||
<th>Type</th>
|
||||
<th>Description</th>
|
||||
<tr>
|
||||
<td>--cacert</td>
|
||||
<td></td>
|
||||
<td>String</td>
|
||||
<td>custom ca certificate used by litmusctl for communicating with portal</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>--config</td>
|
||||
<td></td>
|
||||
<td>String</td>
|
||||
<td>config file (default is $HOME/.litmusctl)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>--skipSSL</td>
|
||||
<td></td>
|
||||
<td>Boolean</td>
|
||||
<td>litmusctl will skip ssl/tls verification while communicating with portal</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>--help</td>
|
||||
<td>-h</td>
|
||||
<td></td>
|
||||
<td>help for litmusctl</td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
For more information related to flags, Use `litmusctl --help`.
|
||||
## Learn More
|
||||
|
||||
- [Learn More about Litmusctl](installation.md)
|
||||
|
|
|
|||
|
|
@ -6,102 +6,87 @@ sidebar_label: Using non interactive mode
|
|||
|
||||
---
|
||||
|
||||
# Usage: Litmusctl v0.10.0
|
||||
# Usage: Litmusctl v0.12.0
|
||||
|
||||
> Notes:
|
||||
>
|
||||
> - For litmusctl v0.10.0 or latest
|
||||
> - Compatible with Litmus 2.9.0 or latest
|
||||
> - For litmusctl v0.12.0 or latest
|
||||
> - Compatible with Litmus 2.12.0 or latest
|
||||
|
||||
### litmusctl Syntax
|
||||
|
||||
`litmusctl` has a syntax to use as follows:
|
||||
|
||||
```shell
|
||||
litmusctl [command] [TYPE] [flags]
|
||||
```
|
||||
|
||||
- Command: refers to what you do want to perform (create, get and config)
|
||||
- Type: refers to the feature type you are performing a command against (chaos delegate, project etc.)
|
||||
- Flags: It takes some additional information for resource operations. For example, `--installation-mode` allows you to specify an installation mode.
|
||||
* Command: refers to what you do want to perform (connect, create, get and config)
|
||||
* Type: refers to the feature type you are performing a command against (chaos-delegate, project etc.)
|
||||
* Flags: It takes some additional information for resource operations. For example, `--installation-mode` allows you to specify an installation mode.
|
||||
|
||||
Litmusctl is using the `.litmusconfig` config file to manage multiple accounts
|
||||
|
||||
1. If the --config flag is set, then only the given file is loaded. The flag may only be set once and no merging takes place.
|
||||
2. Otherwise, the ${HOME}/.litmusconfig file is used, and no merging takes place.
|
||||
|
||||
Litmusctl supports both interactive and non-interactive(flag based) modes.
|
||||
|
||||
> Only `litmusctl connect agent` command needs --non-interactive flag, other commands don't need this flag to be in non-interactive mode. If mandatory flags aren't passed, then litmusctl takes input in an interactive mode.
|
||||
> Only `litmusctl connect chaos-delegate` command needs --non-interactive flag, other commands don't need this flag to be in non-interactive mode. If mandatory flags aren't passed, then litmusctl takes input in an interactive mode.
|
||||
|
||||
### Installation modes
|
||||
Litmusctl can install a Chaos Delegate in two different modes.
|
||||
* cluster mode: With this mode, the Chaos Delegate can run the chaos in any namespace. It installs appropriate cluster roles and cluster role bindings to achieve this mode. It can be enabled by passing a flag `--installation-mode=cluster`
|
||||
|
||||
Litmusctl can install an chaos delegate in two different modes.
|
||||
* namespace mode: With this mode, the Chaos Delegate can run the chaos in its namespace. It installs appropriate roles and role bindings to achieve this mode. It can be enabled by passing a flag `--installation-mode=namespace`
|
||||
|
||||
- cluster mode: With this mode, the chaos delegate can run the chaos in any namespace. It installs appropriate cluster roles and cluster role bindings to achieve this mode. It can be enabled by passing a flag `--installation-mode=cluster`
|
||||
Note: With namespace mode, the user needs to create the namespace to install the Chaos Delegate as a prerequisite.
|
||||
|
||||
- namespace mode: With this mode, the chaos delegate can run the chaos in its namespace. It installs appropriate roles and role bindings to achieve this mode. It can be enabled by passing a flag `--installation-mode=namespace`
|
||||
|
||||
Note: With namespace mode, the user needs to create the namespace to install the chaos delegate as a prerequisite.
|
||||
|
||||
### Minimal steps to connect an chaos delegate
|
||||
|
||||
- To setup an account with litmusctl
|
||||
### Minimal steps to connect a Chaos Delegate
|
||||
|
||||
* To setup an account with litmusctl
|
||||
```shell
|
||||
litmusctl config set-account --endpoint="" --username="" --password=""
|
||||
```
|
||||
|
||||
- To connect a chaos delegate without a project in a cluster mode
|
||||
> Note: If the user doesn't have any project, it will create a random project and add the chaos delegate in that random project.
|
||||
* To create an Chaos Delegate with an existing project
|
||||
> Note: To get `project-id`. Apply `litmusctl get projects`
|
||||
|
||||
```shell
|
||||
litmusctl connect agent --agent-name="" --non-interactive
|
||||
litmusctl connect chaos-delegate --name="" --project-id="" --non-interactive
|
||||
```
|
||||
|
||||
Or,
|
||||
|
||||
- To connect a chaos delegate with an existing project
|
||||
> Note: To get `project-id`. Apply `litmusctl get projects`
|
||||
|
||||
```shell
|
||||
litmusctl connect agent --agent-name="" --project-id="" --non-interactive
|
||||
```
|
||||
|
||||
#### Verify the new Agent Connection
|
||||
|
||||
To verify, if the connection process was successful you can view the list of connected chaos delegate from the Targets section on your ChaosCenter and ensure that the connected chaos delegate is in Active State.
|
||||
|
||||
### Flags for `connect agent` command
|
||||
|
||||
### Flags for `connect chaos-delegate` command
|
||||
<table>
|
||||
<th>Flag</th>
|
||||
<th>Short Flag</th>
|
||||
<th>Type</th>
|
||||
<th>Description</th>
|
||||
<tr>
|
||||
<td>--agent-description</td>
|
||||
<td>--description</td>
|
||||
<td></td>
|
||||
<td>String</td>
|
||||
<td>Set the chaos delegate description (default "---")</td>
|
||||
<td>Set the Chaos Delegate description (default "---")</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>--agent-name</td>
|
||||
<td>--name</td>
|
||||
<td></td>
|
||||
<td>String</td>
|
||||
<td>Set the chaos delegate name</td>
|
||||
<td>Set the name of Chaos Delegate which should be unique</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>--cluster-type</td>
|
||||
<td>--skip-ssl</td>
|
||||
<td></td>
|
||||
<td>Boolean</td>
|
||||
<td>Set whether Chaos Delegate will skip ssl/tls check (can be used for self-signed certs, if cert is not provided in portal) (default false)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>--chaos-delegate-type</td>
|
||||
<td></td>
|
||||
<td>String</td>
|
||||
<td>Set the cluster-type to external for external chaos delegates | Supported=external/internal (default "external")</td>
|
||||
<td>Set the chaos-delegate-type to external for external Chaos Delegates | Supported=external/internal (default "external")</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>--installation-mode</td>
|
||||
<td></td>
|
||||
<td>String</td>
|
||||
<td>Set the installation mode for the kind of chaos delegate | Supported=cluster/namespace (default "cluster")</td>
|
||||
<td>Set the installation mode for the kind of Chaos Delegate | Supported=cluster/namespace (default "cluster")</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>--kubeconfig</td>
|
||||
|
|
@ -113,13 +98,13 @@ To verify, if the connection process was successful you can view the list of con
|
|||
<td>--namespace</td>
|
||||
<td></td>
|
||||
<td>String</td>
|
||||
<td>Set the namespace for the chaos delegate installation (default "litmus")</td>
|
||||
<td>Set the namespace for the Chaos Delegate installation (default "litmus")</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>--node-selector</td>
|
||||
<td></td>
|
||||
<td>String</td>
|
||||
<td>Set the node-selector for chaos delegate components | Format: key1=value1,key2=value2)</td>
|
||||
<td>Set the node-selector for Chaos Delegate components | Format: key1=value1,key2=value2)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>--non-interactive</td>
|
||||
|
|
@ -149,7 +134,7 @@ To verify, if the connection process was successful you can view the list of con
|
|||
<td>--service-account</td>
|
||||
<td></td>
|
||||
<td>String</td>
|
||||
<td>Set the service account to be used by the chaos delegate (default "litmus")</td>
|
||||
<td>Set the service account to be used by the Chaos Delegate (default "litmus")</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>--config</td>
|
||||
|
|
@ -159,16 +144,33 @@ To verify, if the connection process was successful you can view the list of con
|
|||
</tr>
|
||||
</table>
|
||||
|
||||
---
|
||||
|
||||
### Steps to create a chaos scenaro
|
||||
|
||||
* To setup an account with litmusctl
|
||||
```shell
|
||||
litmusctl config set-account --endpoint="" --username="" --password=""
|
||||
```
|
||||
|
||||
* To create a Chaos Scenario by passing a manifest file
|
||||
> Note:
|
||||
> * To get `project-id`, apply `litmusctl get projects`
|
||||
> * To get `chaos-delegate-id`, apply `litmusctl get chaos-delegates --project-id=""`
|
||||
```shell
|
||||
litmusctl create chaos-scenario -f custom-chaos-scenario.yml --project-id="" --chaos-delegate-id=""
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Additional commands
|
||||
|
||||
- To view the current configuration of `.litmusconfig`, type:
|
||||
|
||||
* To view the current configuration of `.litmusconfig`, type:
|
||||
```shell
|
||||
litmusctl config view
|
||||
```
|
||||
|
||||
**Output:**
|
||||
|
||||
```
|
||||
accounts:
|
||||
- users:
|
||||
|
|
@ -185,7 +187,7 @@ current-user: litmus-user
|
|||
kind: Config
|
||||
```
|
||||
|
||||
- To get an overview of the accounts available within `.litmusconfig`, use the `config get-accounts` command:
|
||||
* To get an overview of the accounts available within `.litmusconfig`, use the `config get-accounts` command:
|
||||
|
||||
```shell
|
||||
litmusctl config get-accounts
|
||||
|
|
@ -199,20 +201,17 @@ CURRENT ENDPOINT USERNAME EXPIRESIN
|
|||
* https://preview.litmuschaos.io raj 2021-07-22 14:33:22 +0530 IST
|
||||
```
|
||||
|
||||
- To alter the current account use the `use-account` command with the --endpoint and --username flags:
|
||||
|
||||
* To alter the current account use the `use-account` command with the --endpoint and --username flags:
|
||||
```shell
|
||||
litmusctl config use-account --endpoint="" --username=""
|
||||
```
|
||||
|
||||
- To create a project, apply the following command with the `--name` flag:
|
||||
|
||||
* To create a project, apply the following command with the `--name` flag:
|
||||
```shell
|
||||
litmusctl create project --name=""
|
||||
```
|
||||
|
||||
- To view all the projects with the user, use the `get projects` command.
|
||||
|
||||
* To view all the projects with the user, use the `get projects` command.
|
||||
```shell
|
||||
litmusctl get projects
|
||||
```
|
||||
|
|
@ -225,32 +224,102 @@ PROJECT ID PROJECT NAME CREATEDAT
|
|||
7a4a259a-1ae5-4204-ae83-89a8838eaec3 DevOps Project 2021-07-21 14:39:14 +0530 IST
|
||||
```
|
||||
|
||||
- To get an overview of the chaos delegate available within a project, issue the following command.
|
||||
|
||||
* To get an overview of the Chaos Delegates available within a project, issue the following command.
|
||||
```shell
|
||||
litmusctl get agents --project-id=""
|
||||
litmusctl get chaos-delegates --project-id=""
|
||||
```
|
||||
|
||||
**Output:**
|
||||
|
||||
```
|
||||
AGENTID AGENTNAME STATUS REGISTRATION
|
||||
55ecc7f2-2754-43aa-8e12-6903e4c6183a agent-1 ACTIVE REGISTERED
|
||||
13dsf3d1-5324-54af-4g23-5331g5v2364f agent-2 INACTIVE NOT REGISTERED
|
||||
CHAOS DELEGATE ID CHAOS DELEGATE NAME STATUS REGISTRATION
|
||||
55ecc7f2-2754-43aa-8e12-6903e4c6183a chaos-delegate-1 ACTIVE REGISTERED
|
||||
13dsf3d1-5324-54af-4g23-5331g5v2364f chaos-delegate-2 INACTIVE NOT REGISTERED
|
||||
```
|
||||
|
||||
- To disconnect a chaos delegate, issue the following command.
|
||||
|
||||
* To disconnect a Chaos Delegate, issue the following command..
|
||||
```shell
|
||||
litmusctl disconnect agent 55ecc7f2-2754-43aa-8e12-6903e4c6183a --project-id=""
|
||||
litmusctl disconnect chaos-delegate <chaos-delegate-id> --project-id=""
|
||||
```
|
||||
|
||||
**Output:**
|
||||
|
||||
```
|
||||
🚀 ChaosAgent successfully disconnected.
|
||||
🚀 Chaos Delegate successfully disconnected.
|
||||
```
|
||||
|
||||
|
||||
* To list the created Chaos Scenarios within a project, issue the following command.
|
||||
```shell
|
||||
litmusctl get chaos-scenarios --project-id=""
|
||||
```
|
||||
|
||||
**Output:**
|
||||
|
||||
```
|
||||
CHAOS SCENARIO ID CHAOS SCENARIO NAME CHAOS SCENARIO TYPE NEXT SCHEDULE CHAOS DELEGATE ID CHAOS DELEGATE NAME LAST UPDATED BY
|
||||
9433b48c-4ab7-4544-8dab-4a7237619e09 custom-chaos-scenario-1627980541 Non Cron Chaos Scenario None f9799723-29f1-454c-b830-ae8ba7ee4c30 Self-Chaos-delegate admin
|
||||
|
||||
Showing 1 of 1 Chaos Scenarios
|
||||
```
|
||||
|
||||
|
||||
* To list all the Chaos Scenario runs within a project, issue the following command.
|
||||
```shell
|
||||
litmusctl get chaos-scenario-runs --project-id=""
|
||||
```
|
||||
|
||||
**Output:**
|
||||
|
||||
```
|
||||
CHAOS SCENARIO RUN ID STATUS RESILIENCY SCORE CHAOS SCENARIO ID CHAOS SCENARIO NAME TARGET CHAOS DELEGATE LAST RUN EXECUTED BY
|
||||
8ceb712c-1ed4-40e6-adc4-01f78d281506 Running 0.00 9433b48c-4ab7-4544-8dab-4a7237619e09 custom-chaos-scenario-1627980541 Self-Chaos-Delegate June 1 2022, 10:28:02 pm admin
|
||||
|
||||
Showing 1 of 1 Chaos Scenario runs
|
||||
```
|
||||
|
||||
|
||||
* To describe a particular Chaos Scenario, issue the following command.
|
||||
```shell
|
||||
litmusctl describe chaos-scenario <chaos-scenario-id> --project-id=""
|
||||
```
|
||||
|
||||
**Output:**
|
||||
|
||||
```
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: Workflow
|
||||
metadata:
|
||||
creationTimestamp: null
|
||||
labels:
|
||||
cluster_id: f9799723-29f1-454c-b830-ae8ba7ee4c30
|
||||
subject: custom-chaos-scenario_litmus
|
||||
workflow_id: 9433b48c-4ab7-4544-8dab-4a7237619e09
|
||||
workflows.argoproj.io/controller-instanceid: f9799723-29f1-454c-b830-ae8ba7ee4c30
|
||||
name: custom-chaos-scenario-1627980541
|
||||
namespace: litmus
|
||||
spec:
|
||||
...
|
||||
```
|
||||
|
||||
|
||||
* To delete a particular Chaos Scenario, issue the following command.
|
||||
```shell
|
||||
litmusctl delete chaos-scenario <chaos-scenario-id> --project-id=""
|
||||
```
|
||||
|
||||
**Output:**
|
||||
|
||||
```
|
||||
🚀 Chaos Scenario successfully deleted.
|
||||
```
|
||||
|
||||
|
||||
|
||||
For more information related to flags, Use `litmusctl --help`.
|
||||
|
||||
## Learn More
|
||||
|
||||
- [Learn More about Litmusctl](installation.md)
|
||||
|
|
|
|||
|
|
@ -0,0 +1,17 @@
|
|||
---
|
||||
id: architecture-summary
|
||||
title: Architecture Summary
|
||||
sidebar_label: Architecture Summary
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
<img src={require("../assets/architecture-summary.png").default} alt="Architecture Overview" />
|
||||
|
||||
The Litmus architecture can be segregated into two parts:
|
||||
|
||||
1. **Control Plane:** Contains the components required for the functioning of Chaos Center, the website-based portal for Litmus.
|
||||
|
||||
2. **Execution Plane:** Contains the components required for the injection of chaos in the target resources.
|
||||
|
||||
Chaos Center can be used for creating, scheduling, and monitoring Chaos Scenarios, a set of chaos experiments defined in a definitive sequence to achieve desired chaos impact on the target resources upon execution. Users can log in to the Chaos Center using valid login credentials and leverage the interactive web UI to define their chaos scenario to target multiple aspects of their infrastructure. Once the user creates a Chaos Scenario using the Chaos Center, it is passed on to the Execution Plane. The Execution Plane can be present either in the host cluster containing the Control Plane if the self chaos delegate is being used, or in the target cluster if an external chaos delegate is being used. The Execution Plane interprets the Chaos Scenario as a list of steps required for injecting chaos into the target resources. It ensures efficient orchestration of chaos in cloud-native environments using various Kubernetes CRs. Once the Chaos Scenario is executed, Execution Plane sends the chaos result to the Control Plane for their post-processing using either the built-in monitoring dashboard of Litmus or using external observability tools such as Prometheus DB and Grafana dashboard. Litmus also achieves automated Chaos Scenario runs to execute chaos as part of the CI/CD pipeline based on a set of defined conditions using GitOps.
|
||||
|
|
@ -0,0 +1,40 @@
|
|||
---
|
||||
id: chaos-control-plane
|
||||
title: Chaos Control Plane
|
||||
sidebar_label: Chaos Control Plane
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
<img src={require("../assets/chaos-control-plane.png").default} alt="Chaos Control Plane" />
|
||||
|
||||
Chaos Control Plane consists of micro-services responsible for the functioning of the Chaos Center, the website-based portal that can be used for interacting with Litmus, apart from the CLI. Chaos Plane facilitates the creation and scheduling of chaos scenarios, system observability during the event of chaos, and post-processing and analysis of experiment results.
|
||||
|
||||
## Chaos Control Plane Components
|
||||
|
||||
- **Authentication Server:** A Golang micro-service that is responsible for authorizing, authenticating the requests received from Chaos Center and managing users along with their projects. It primarily serves the cause of user creation, user login, resetting the password, updating user information, creating project, managing project related operations.
|
||||
|
||||
- **Backend Server:** A GraphQL based Golang micro-service that serves the requests received from Chaos Center, by either querying the database for the relevant information or by fetching information from the Execution Plane.
|
||||
|
||||
- **Database:** A NoSQL MongoDB database micro-service that is accountable for storing users' information, past chaos scenarios, saved chaos scenario templates, user projects, ChaosHubs, and GitOps details, among the other information.
|
||||
|
||||
- **Chaos Center:** Refers to the interfaces used by Litmus for creation and scheduling of chaos scenarios, system observability during chaos injection, and post chaos result analysis. It includes:
|
||||
|
||||
- **Web UI:** A React.js based frontend application micro-service with built-in system observability capabilities and an analytics dashboard. It also facilitates teams of users to collaborate over chaos scenarios using role-based user accounts.
|
||||
|
||||
- **Litmusctl:** A command-line tool that allows management of Litmus Chaos Delegate Infrastructure components. It can be used to create chaos delegates, project, and manage multiple Litmus accounts.
|
||||
|
||||
- **Litmus API:** Refers to two different Litmus APIs, namely Litmus Authentication API and Litmus Portal API:
|
||||
|
||||
- **Litmus Authentication API:** Used to authenticate the identity of a user and to perform several user and project specific tasks like create new users, update profile, update password, create project, invite users to project, get project details etc. It uses the Authentication Server to perform these tasks.
|
||||
|
||||
- **Litmus Portal API:** Provides command-line and UI experience for managing and monitoring the events around chaos scenarios. It uses the Backend Server to perform its functions.
|
||||
|
||||
## Standard Chaos Control Plane Flow
|
||||
|
||||
1. The User logs in to the ChaosCenter using a valid login credential. A default project is created for the user on initial login. Every user is a part of a project and has a role assigned to them. To schedule a chaos scenario, the user needs to have an Editor or Owner role assigned in the project.
|
||||
2. The user uploads a Chaos Scenario manifest using the ChaosCenter, which is received by the Backend Server.
|
||||
3. Backend Server stores the manifest in the Database and also sends it to the Chaos Delegate.
|
||||
4. Chaos Delegate uses the Chaos Scenario manifest to inject chaos into the target resources. The steps of the Chaos Scenario execution can be visualized using the ChaosCenter.
|
||||
5. Chaos Delegate returns the results of the chaos experiments that were a part of the chaos scenario back to the Backend Server, along with the experiment logs.
|
||||
6. Backend Server then sends the chaos experiment results and logs to the ChaosCenter. It also stores the results into the Database for generating post-chaos scenario statistics and information.
|
||||
|
|
@ -0,0 +1,64 @@
|
|||
---
|
||||
id: chaos-execution-plane
|
||||
title: Chaos Execution Plane
|
||||
sidebar_label: Chaos Execution Plane
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
<img src={require("../assets/chaos-execution-plane.png").default} alt="Chaos Execution Plane" />
|
||||
|
||||
Chaos Execution Plane contains the components responsible for orchestrating the chaos injection in the target resources. They get installed in either an external target cluster if an external chaos delegate is being used or in the host cluster containing the control plane if a self chaos delegate is being used. It can be further segregated into Litmus Chaos Delegate Infrastructure components and Litmus Backend Execution Infrastructure components.
|
||||
|
||||
## Litmus Execution Plane Components
|
||||
|
||||
Litmus Chaos Delegate Infrastructure components help facilitate the chaos injection, manage chaos observability, and enable chaos automation for target resources. These components include:
|
||||
|
||||
1. **Workflow Controller:** The Argo Workflow Controller responsible for the creation of Chaos Scenarios using the Chaos Scenario CR.
|
||||
|
||||
2. **Subscriber:** Serves as the link between the Chaos Execution Plane and the Control Plane. It has a few distinct responsibilities such as performing health check of all the components in Chaos Execution Plane, creation of a Chaos Scenario CR from a Chaos Scenario template, watching for Chaos Scenario events during its execution, and sending the chaos scenario result to the Control Plane.
|
||||
|
||||
3. **Event Tracker:** An optional component that is capable of triggering automated chaos scenario runs based on a set of defined conditions for any given resources in the cluster. It is a controller that manages EventTrackerPolicy CR, which is basically the set of defined conditions that is validated by Event Tracker. If the current state of the tracked resources match with the state defined in the EventTrackerPolicy CR, the chaos scenario run run gets triggered. This feature can only be used if GitOps is enabled.
|
||||
|
||||
4. **Chaos Exporter:** An optional component that facilitates external observability in Litmus by exporting the chaos metrics generated during the chaos injection as time-series data to the Prometheus DB for its processing and analysis.
|
||||
|
||||
Litmus Backend Execution Infrastructure components orchestrate the execution of Chaos Scenario in target resources. These components include:
|
||||
|
||||
1. **Chaos Scenario CR:** Refers to the Argo Workflow CR which describes the steps that are executed as a part of the chaos scenario. It is used to define failures during a certain workload condition (such as, say, percentage load), multiple (parallel) failures of dependent and independent services etc.
|
||||
|
||||
2. **ChaosExperiment CR:** Used for defining the low-level execution information for any Litmus chaos experiment as well as to store the various experiment tunables.
|
||||
|
||||
3. **ChaosEngine CR:** Used to hold information about how the chaos experiments are executed. It connects an application instance with one or more chaos experiments while allowing the users to specify run-level details.
|
||||
|
||||
4. **Chaos Operator:** A Kubernetes custom-controller that manages the lifecycle of certain resources or applications intending to validate their "desired state". It helps reconcile the state of the ChaosEngine by performing specific actions upon CRUD of the ChaosEngine. It also defines a secondary resource (the ChaosEngine Runner pod), which is created & managed by it to implement the reconcile functions.
|
||||
|
||||
<div style={{textAlign: 'center'}}>
|
||||
<img src={require("../assets/chaos-execution-plane-chaos-operator.png").default} alt="Chaos Operator" />
|
||||
</div>
|
||||
|
||||
5. **ChaosResult CR:** Holds the results of a chaos experiment, such as ChaosEngine reference, Experiment State, Verdict of the experiment (on completion), salient application/result attributes. It also acts as a source for metrics collection for observability.
|
||||
|
||||
6. **Chaos Runner:** Acts as a bridge between the Chaos Operator and Chaos Experiments. It is a lifecycle manager for the chaos experiments that creates Experiment Jobs for the execution of experiment business logic and monitors the experiment pods (jobs) until completion.
|
||||
|
||||
<div style={{textAlign: 'center'}}>
|
||||
<img src={require("../assets/chaos-execution-plane-chaos-runner.png").default} alt="Chaos Runner" />
|
||||
</div>
|
||||
|
||||
7. **Experiment Jobs:** Refers to the pods that execute the experiment logic. One experiment pod is created per chaos experiment in the chaos scenario.
|
||||
|
||||
## Standard Chaos Execution Plane Flow
|
||||
|
||||
1. Subscriber receives the Chaos Scenario manifest from the Control Plane and applies the manifest to create a Chaos Scenario CR.
|
||||
2. Chaos Scenario CRs are tracked by the Argo Workflow Controller. When the Workflow Controller finds a new Chaos Scenario CR, it creates the ChaosExperiment CRs and the ChaosEngine CRs for the chaos experiments that are a part of the chaos scenario.
|
||||
3. ChaosEngine CRs are tracked by the Chaos Operator. Once a ChaosEngine CR is ready, the Chaos Operator updates the ChaosEngine state to reflect that the particular ChaosEngine is now being executed.
|
||||
4. For each ChaosEngine resource, a Chaos Runner is created by the Chaos Operator.
|
||||
5. Chaos Runner firstly reads the chaos parameters from the ChaosExperiment CR and overrides them with values from the ChaosEngine CR. It then constructs the Experiment Jobs and monitors them until their completion.
|
||||
6. Experiment Jobs execute the experiment business logic and undertake chaos injection on target resources. Once done, the ChaosResult is updated with the experiment verdict.
|
||||
7. Chaos Runner then fetches the updated ChaosResult and updates the ChaosEngine status as well as the verdict.
|
||||
8. Once the ChaosEngine is updated, Subscriber fetches the ChaosEngine details and the ChaosResult and forwards them to Chaos Control Plane.
|
||||
|
||||
It is worth noticing that:
|
||||
|
||||
- If configured, Chaos Exporter fetches data from the ChaosResult CR and converts it in a time-series format to be consumed by the Prometheus DB.
|
||||
|
||||
- An Event Tracker Policy can also be set up as part of the Backend GitOps, where the Backend GitOps Controller tracks a set of specified resources in the target cluster for any change. If any of the tracked resources undergo any change and their resulting state matches the state defined in the Event Tracker Policy, then a pre-defined Chaos Scenario is executed.
|
||||
|
|
@ -0,0 +1,30 @@
|
|||
---
|
||||
id: chaos-experiment-flow
|
||||
title: Chaos Experiment Flow
|
||||
sidebar_label: Chaos Experiment Flow
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
<img src={require("../assets/experiment-flow.png").default} alt="Chaos Experiment Flow" />
|
||||
|
||||
The experiment execution is triggered upon the creation of a ChaosEngine resource. The ChaosEngine resource interacts with Chaos Runner, which is created by the Chaos Operator. The Chaos Runner creates Experiment Jobs that execute the experiment business logic. Typically, these ChaosEngines are embedded within the 'steps' of a Litmus Chaos Scenario. However, one may also create and apply the Chaos Engines manually, and then the chaos-operator reconciles this resource and triggers the experiment execution. Chaos experiments are classified as:
|
||||
|
||||
- Kubernetes Experiments
|
||||
- Pod-Level Chaos
|
||||
- Node-Level Chaos
|
||||
- Application Chaos
|
||||
- Cloud Infrastructure
|
||||
|
||||
## Chaos Experiment Flow Steps
|
||||
|
||||
1. Chaos experiment execution gets triggered by the Experiment Job.
|
||||
2. Experiment tunables and low-level execution details are fetched.
|
||||
3. ChaosResult gets initialized and its verdict is updated as "Awaited" to indicate that the experiment is currently running.
|
||||
4. Steady-state condition for the respective experiment is validated. If the condition is found to be invalid, the experiment execution is stopped and the ChaosResult is updated as "Fail".
|
||||
5. Once the steady-state condition is validated, experiment resources are created to facilitate the chaos injection.
|
||||
6. Chaos injection is performed on the target resources for the specified chaos duration.
|
||||
7. Chaos injection gets reverted.
|
||||
8. Post chaos status-check is performed to ensure that the steady-state is still maintained.
|
||||
9. If the check is invalid, the ChaosEngine and ChaosResult verdicts are updated as "Fail", otherwise they are updated as "Pass".
|
||||
10. Experiment execution ends.
|
||||
|
|
@ -0,0 +1,23 @@
|
|||
---
|
||||
id: chaos-observability-flow-analytics
|
||||
title: Analytics
|
||||
sidebar_label: Analytics
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
<img src={require("../assets/chaos-observability-flow-analytics.png").default} alt="Chaos Observability Flow Analytics" />
|
||||
|
||||
Analytics is an integral part of Chaos Engineering, as it offers key insights that are required to fully understand a system during the chaos and functions as a decision-making tool for improving system resiliency.
|
||||
|
||||
In Litmus, chaos scenario run statistics and information are generated post the chaos scenario execution, which can be accessed directly using the ChaosCenter.
|
||||
|
||||
## Observability Flow for Analytics
|
||||
|
||||
1. In the Chaos Execution Plane, the ChaosEngine Details and ChaosResult are fetched by the Chaos Delegate.
|
||||
2. Chaos Delegate then forwards them to the Backend Server in the Chaos Control Plane and later they get stored into the Database.
|
||||
3. User specifies the Chaos Scenario Schedule for which the Chaos Scenario statistics and information is to be fetched as an input in the ChaosCenter.
|
||||
4. The request for the Chaos Scenario statistics and information is received by the Backend Server.
|
||||
5. Backend Server queries the Database for the details of past Chaos Scenario Runs.
|
||||
6. Aggregated Chaos Scenario statistics based on the ChaosResult verdict and probe success percentage are fetched from the Database by Backend Server.
|
||||
7. Chaos Scenario statistics and information are forwarded to ChaosCenter by Backend Server.
|
||||
|
|
@ -0,0 +1,23 @@
|
|||
---
|
||||
id: chaos-observability-flow-logging
|
||||
title: Logging
|
||||
sidebar_label: Logging
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
<img src={require("../assets/chaos-observability-flow-logging.png").default} alt="Chaos Observability Flow Logging" />
|
||||
|
||||
Logging is a pivotal observability aspect in LitmusChaos as it allows the user to track the exact system behavior during the scenario of a chaos. The logs can be classified into one of the following:
|
||||
|
||||
- **Litmus Checker Logs:** Logs generated as part of the validation for chaos resources that are required to execute a chaos experiment.
|
||||
- **Experiment Logs:** Logs generated as part of the steps performed during the chaos experiment, including pre-chaos check logs, chaos injection logs, chaos probes logs, and post-chaos check logs.
|
||||
- **Non-Chaos Scenario Step Logs:** Logs generated as part of the chaos scenario steps that facilitate the execution of the chaos experiment, such as chaos experiment installation step logs, chaos revert step logs, etc.
|
||||
|
||||
## Observability Flow for Logging
|
||||
|
||||
1. User requests the logs for any particular chaos scenario step using the ChaosCenter.
|
||||
2. The request for the logs is received by the Backend Server and is forwarded to the Subscriber.
|
||||
3. The subscriber checks if the chaos scenario step is a Chaos Experiment step or not.
|
||||
4. If the chaos scenario step is a Chaos Experiment step, then the Litmus Checker logs and the Chaos Experiment Logs are fetched from the ChaosEngine CR by the subscriber. Else, the logs of the chaos scenario step pod is fetched from the respective chaos scenario step pod by the subscriber.
|
||||
5. The fetched logs are returned to the Backend Server, which returns them to the ChaosCenter.
|
||||
|
|
@ -0,0 +1,38 @@
|
|||
---
|
||||
id: chaos-observability-flow-monitoring
|
||||
title: Monitoring
|
||||
sidebar_label: Monitoring
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
Monitoring is a key part to effectively watch and understand the state of the systems. In Litmus, the monitoring is split into a two-fold approach:
|
||||
- Metrics Monitoring
|
||||
- Events Monitoring
|
||||
|
||||
## Metrics Monitoring
|
||||
|
||||
<div style={{textAlign: 'center'}}>
|
||||
<img src={require("../assets/chaos-observability-flow-metrics.png").default} alt="Chaos Observability Flow Metrics" />
|
||||
</div>
|
||||
|
||||
Metrics Monitoring enables the users to monitor the chaos metrics generated during chaos injection, which is exported by the chaos exporter to be consumed as time-series information.
|
||||
|
||||
### Observability Flow for Metrics Monitoring
|
||||
1. During the event of chaos, the ChaosResult CR and the ChaosEngine CR are fetched by the Chaos Exporter.
|
||||
2. If ChaosResult verdict is "Awaited", then the Continuous Event Metrics are fetched by Chaos Exporter. Else, if the verdict is "Pass" or "Fail" or "Stopped", then the Gauge Metrics are fetched by Chaos Exporter at a fixed TSDB Scrape Interval.
|
||||
3. The fetched metrics are then exposed at the Chaos Exporter Kubernetes service.
|
||||
4. TSDBs consume these metrics and store them as time-series value.
|
||||
5. APMs and Visualisation Tools query and fetch the chaos metric from the TSDBs.
|
||||
|
||||
## Events Monitoring
|
||||
|
||||
<div style={{textAlign: 'center'}}>
|
||||
<img src={require("../assets/chaos-observability-flow-events.png").default} alt="Chaos Observability Flow Events" />
|
||||
</div>
|
||||
|
||||
Events Monitoring enables the users to monitor the Kubernetes events that are created as part of the orchestration of chaos injection by Litmus. These events occur through different points in the lifetime of a Chaos Experiment's execution to accomplish.
|
||||
|
||||
- Like any other Kubernetes Events, these events also get stored inside the etcd.
|
||||
- The ChaosEngine CR events are initiated by the Operator or the Chaos Runner or the Chaos Experiment itself.
|
||||
- The ChaosResult CR events are initiated by the Chaos Experiment itself.
|
||||
|
|
@ -0,0 +1,38 @@
|
|||
---
|
||||
id: chaos-observability-flow-overview
|
||||
title: Overview
|
||||
sidebar_label: Overview
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
Observability in Litmus serves a two-fold cause:
|
||||
|
||||
1. To provide the right hooks to APM platforms so as to enable visualization and understand the behavior of application/microservices under chaotic conditions.
|
||||
|
||||
2. Ability to gather, record & factor in data provided by standard observability frameworks as part of SLO validation in automated chaos experiment runs - the results of which can be stored & analyzed as experiment “verdicts” or “metadata”.
|
||||
|
||||
Chaos Observability in Litmus can be sectioned into the following:
|
||||
|
||||
1. **[Visualising Chaos Scenario (Visualization)](chaos-observability-flow-visualization.md)**
|
||||
|
||||
- Chaos Scenario Execution Graph
|
||||
|
||||
2. **[Fetching Logs (Logging)](chaos-observability-flow-logging.md)**
|
||||
|
||||
- Litmus Checker Logs
|
||||
- Experiment Logs
|
||||
- Non-Chaos Scenario Logs
|
||||
|
||||
3. **[Monitoring Systems in Real Time During Chaos (Monitoring)](chaos-observability-flow-monitoring.md)**
|
||||
|
||||
- Metrics
|
||||
- Events
|
||||
|
||||
4. **[Viewing Experiment Verdict and Summary (Summarisation)](chaos-observability-flow-summarisation.md)**
|
||||
|
||||
- Chaos Result
|
||||
|
||||
5. **[Post-Chaos Scenario Analytics (Analytics)](chaos-observability-flow-analytics.md)**
|
||||
|
||||
- Chaos Scenario Statistics and Information
|
||||
|
|
@ -0,0 +1,22 @@
|
|||
---
|
||||
id: chaos-observability-flow-summarisation
|
||||
title: Summarisation
|
||||
sidebar_label: Summarisation
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
<img src={require("../assets/chaos-observability-flow-summarisation.png").default} alt="Chaos Observability Flow Summarisation" />
|
||||
|
||||
Summarisation is the process of aggregating the information pertaining to a Chaos Experiment's execution over time.
|
||||
|
||||
In Litmus, summarisation is performed using the ChaosResult CR. It stores all the information relevant to a ChaosExperiment's execution over time, including events, experiment verdict, probe success percentage, target resource details, etc.
|
||||
|
||||
## Observability Flow for Summarisation
|
||||
|
||||
1. In the Chaos Execution Plane, the ChaosEngine Details and Chaos Result are fetched by the Chaos Delegate.
|
||||
2. Chaos Delegate then forwards them to the Backend Server in the Chaos Control Plane and later they get stored into the Database.
|
||||
3. When a user requests for the ChaosResult for a particular Chaos Experiment using the ChaosCenter, the request is received by the Backend Server.
|
||||
4. Backend Server queries the Database for the particular ChaosResult for the input Chaos Experiment.
|
||||
5. The ChaosResult is fetched from the Database by the Backend Server.
|
||||
6. ChaosResult is forwarded to the ChaosCenter.
|
||||
|
|
@ -0,0 +1,15 @@
|
|||
---
|
||||
id: chaos-observability-flow-visualization
|
||||
title: Visualization
|
||||
sidebar_label: Visualization
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
<img src={require("../assets/argo-graph.png").default} alt="Chaos Observability Flow Visualisation" />
|
||||
|
||||
Chaos scenario visualization constitutes an integral part of the system observability during chaos in Litmus.
|
||||
|
||||
- It is a visual representation of the steps performed as part of the Chaos Scenario CR execution.
|
||||
- Every step is represented as a node, which also depicts the status of the respective node that it represents.
|
||||
- All the nodes are connected in a sequential order to account for the order of execution of the respective steps.
|
||||
|
|
@ -0,0 +1,29 @@
|
|||
---
|
||||
id: overview
|
||||
title: Overview
|
||||
sidebar_label: Overview
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
The Architecture section contains the component overview, sequence diagrams and description of flow of information through the Litmus architecture.
|
||||
|
||||
### [Architecture Summary](architecture-summary.md)
|
||||
|
||||
A very high level overview of the entire Litmus architecture with the objective of highlighting the flow of information through the various components.
|
||||
|
||||
### [Control Plane](chaos-control-plane.md)
|
||||
|
||||
Consists of micro-services responsible for the functioning of the ChaosCenter, the web based portal used for creating, scheduling, and monitoring chaos scenarios.
|
||||
|
||||
### [Execution Plane](chaos-execution-plane.md)
|
||||
|
||||
Contains the components required for the orchestration of chaos injection in the target resources.
|
||||
|
||||
### [Chaos Experiment Flow](chaos-experiment-flow.md)
|
||||
|
||||
Flow of information during the execution of Litmus chaos experiments, grouped into the categories such as pod-level, node-level, application-level, and public-cloud.
|
||||
|
||||
### [Chaos Observability Flow](chaos-observability-flow-overview.md)
|
||||
|
||||
Information flow for monitoring and observability during and post chaos using built-in Litmus analytics dashboard as well as external observability tools.
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 655 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 33 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 181 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 134 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 281 B |
File diff suppressed because one or more lines are too long
Binary file not shown.
|
After Width: | Height: | Size: 161 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 87 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 56 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 211 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 52 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 90 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 34 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 104 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 74 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 78 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 57 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 112 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 106 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 124 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 194 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 114 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 195 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 151 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 123 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 56 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 53 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 189 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 31 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 23 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 67 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 115 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 334 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 15 KiB |
|
|
@ -0,0 +1,429 @@
|
|||
<?xml version="1.0"?>
|
||||
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" width="1349.33333328" height="862.4" font-family="Consolas, Menlo, 'Bitstream Vera Sans Mono', monospace, 'Powerline Symbols'" font-size="14px">
|
||||
<style>
|
||||
<!-- asciinema theme -->
|
||||
|
||||
.default-text-fill {fill: #cccccc}
|
||||
.default-bg-fill {fill: #121314}
|
||||
|
||||
.c-0 {fill: #000000}
|
||||
.c-1 {fill: #dd3c69}
|
||||
.c-2 {fill: #4ebf22}
|
||||
.c-3 {fill: #ddaf3c}
|
||||
.c-4 {fill: #26b0d7}
|
||||
.c-5 {fill: #b954e1}
|
||||
.c-6 {fill: #54e1b9}
|
||||
.c-7 {fill: #d9d9d9}
|
||||
.c-8 {fill: #4d4d4d}
|
||||
.c-9 {fill: #dd3c69}
|
||||
.c-10 {fill: #4ebf22}
|
||||
.c-11 {fill: #ddaf3c}
|
||||
.c-12 {fill: #26b0d7}
|
||||
.c-13 {fill: #b954e1}
|
||||
.c-14 {fill: #54e1b9}
|
||||
.c-15 {fill: #ffffff}
|
||||
.c-8, .c-9, .c-10, .c-11, .c-12, .c-13, .c-14, .c-15 {font-weight: bold}
|
||||
<!-- 256 colors -->
|
||||
|
||||
.c-16 {fill: #000000}
|
||||
.c-17 {fill: #00005f}
|
||||
.c-18 {fill: #000087}
|
||||
.c-19 {fill: #0000af}
|
||||
.c-20 {fill: #0000d7}
|
||||
.c-21 {fill: #0000ff}
|
||||
.c-22 {fill: #005f00}
|
||||
.c-23 {fill: #005f5f}
|
||||
.c-24 {fill: #005f87}
|
||||
.c-25 {fill: #005faf}
|
||||
.c-26 {fill: #005fd7}
|
||||
.c-27 {fill: #005fff}
|
||||
.c-28 {fill: #008700}
|
||||
.c-29 {fill: #00875f}
|
||||
.c-30 {fill: #008787}
|
||||
.c-31 {fill: #0087af}
|
||||
.c-32 {fill: #0087d7}
|
||||
.c-33 {fill: #0087ff}
|
||||
.c-34 {fill: #00af00}
|
||||
.c-35 {fill: #00af5f}
|
||||
.c-36 {fill: #00af87}
|
||||
.c-37 {fill: #00afaf}
|
||||
.c-38 {fill: #00afd7}
|
||||
.c-39 {fill: #00afff}
|
||||
.c-40 {fill: #00d700}
|
||||
.c-41 {fill: #00d75f}
|
||||
.c-42 {fill: #00d787}
|
||||
.c-43 {fill: #00d7af}
|
||||
.c-44 {fill: #00d7d7}
|
||||
.c-45 {fill: #00d7ff}
|
||||
.c-46 {fill: #00ff00}
|
||||
.c-47 {fill: #00ff5f}
|
||||
.c-48 {fill: #00ff87}
|
||||
.c-49 {fill: #00ffaf}
|
||||
.c-50 {fill: #00ffd7}
|
||||
.c-51 {fill: #00ffff}
|
||||
.c-52 {fill: #5f0000}
|
||||
.c-53 {fill: #5f005f}
|
||||
.c-54 {fill: #5f0087}
|
||||
.c-55 {fill: #5f00af}
|
||||
.c-56 {fill: #5f00d7}
|
||||
.c-57 {fill: #5f00ff}
|
||||
.c-58 {fill: #5f5f00}
|
||||
.c-59 {fill: #5f5f5f}
|
||||
.c-60 {fill: #5f5f87}
|
||||
.c-61 {fill: #5f5faf}
|
||||
.c-62 {fill: #5f5fd7}
|
||||
.c-63 {fill: #5f5fff}
|
||||
.c-64 {fill: #5f8700}
|
||||
.c-65 {fill: #5f875f}
|
||||
.c-66 {fill: #5f8787}
|
||||
.c-67 {fill: #5f87af}
|
||||
.c-68 {fill: #5f87d7}
|
||||
.c-69 {fill: #5f87ff}
|
||||
.c-70 {fill: #5faf00}
|
||||
.c-71 {fill: #5faf5f}
|
||||
.c-72 {fill: #5faf87}
|
||||
.c-73 {fill: #5fafaf}
|
||||
.c-74 {fill: #5fafd7}
|
||||
.c-75 {fill: #5fafff}
|
||||
.c-76 {fill: #5fd700}
|
||||
.c-77 {fill: #5fd75f}
|
||||
.c-78 {fill: #5fd787}
|
||||
.c-79 {fill: #5fd7af}
|
||||
.c-80 {fill: #5fd7d7}
|
||||
.c-81 {fill: #5fd7ff}
|
||||
.c-82 {fill: #5fff00}
|
||||
.c-83 {fill: #5fff5f}
|
||||
.c-84 {fill: #5fff87}
|
||||
.c-85 {fill: #5fffaf}
|
||||
.c-86 {fill: #5fffd7}
|
||||
.c-87 {fill: #5fffff}
|
||||
.c-88 {fill: #870000}
|
||||
.c-89 {fill: #87005f}
|
||||
.c-90 {fill: #870087}
|
||||
.c-91 {fill: #8700af}
|
||||
.c-92 {fill: #8700d7}
|
||||
.c-93 {fill: #8700ff}
|
||||
.c-94 {fill: #875f00}
|
||||
.c-95 {fill: #875f5f}
|
||||
.c-96 {fill: #875f87}
|
||||
.c-97 {fill: #875faf}
|
||||
.c-98 {fill: #875fd7}
|
||||
.c-99 {fill: #875fff}
|
||||
.c-100 {fill: #878700}
|
||||
.c-101 {fill: #87875f}
|
||||
.c-102 {fill: #878787}
|
||||
.c-103 {fill: #8787af}
|
||||
.c-104 {fill: #8787d7}
|
||||
.c-105 {fill: #8787ff}
|
||||
.c-106 {fill: #87af00}
|
||||
.c-107 {fill: #87af5f}
|
||||
.c-108 {fill: #87af87}
|
||||
.c-109 {fill: #87afaf}
|
||||
.c-110 {fill: #87afd7}
|
||||
.c-111 {fill: #87afff}
|
||||
.c-112 {fill: #87d700}
|
||||
.c-113 {fill: #87d75f}
|
||||
.c-114 {fill: #87d787}
|
||||
.c-115 {fill: #87d7af}
|
||||
.c-116 {fill: #87d7d7}
|
||||
.c-117 {fill: #87d7ff}
|
||||
.c-118 {fill: #87ff00}
|
||||
.c-119 {fill: #87ff5f}
|
||||
.c-120 {fill: #87ff87}
|
||||
.c-121 {fill: #87ffaf}
|
||||
.c-122 {fill: #87ffd7}
|
||||
.c-123 {fill: #87ffff}
|
||||
.c-124 {fill: #af0000}
|
||||
.c-125 {fill: #af005f}
|
||||
.c-126 {fill: #af0087}
|
||||
.c-127 {fill: #af00af}
|
||||
.c-128 {fill: #af00d7}
|
||||
.c-129 {fill: #af00ff}
|
||||
.c-130 {fill: #af5f00}
|
||||
.c-131 {fill: #af5f5f}
|
||||
.c-132 {fill: #af5f87}
|
||||
.c-133 {fill: #af5faf}
|
||||
.c-134 {fill: #af5fd7}
|
||||
.c-135 {fill: #af5fff}
|
||||
.c-136 {fill: #af8700}
|
||||
.c-137 {fill: #af875f}
|
||||
.c-138 {fill: #af8787}
|
||||
.c-139 {fill: #af87af}
|
||||
.c-140 {fill: #af87d7}
|
||||
.c-141 {fill: #af87ff}
|
||||
.c-142 {fill: #afaf00}
|
||||
.c-143 {fill: #afaf5f}
|
||||
.c-144 {fill: #afaf87}
|
||||
.c-145 {fill: #afafaf}
|
||||
.c-146 {fill: #afafd7}
|
||||
.c-147 {fill: #afafff}
|
||||
.c-148 {fill: #afd700}
|
||||
.c-149 {fill: #afd75f}
|
||||
.c-150 {fill: #afd787}
|
||||
.c-151 {fill: #afd7af}
|
||||
.c-152 {fill: #afd7d7}
|
||||
.c-153 {fill: #afd7ff}
|
||||
.c-154 {fill: #afff00}
|
||||
.c-155 {fill: #afff5f}
|
||||
.c-156 {fill: #afff87}
|
||||
.c-157 {fill: #afffaf}
|
||||
.c-158 {fill: #afffd7}
|
||||
.c-159 {fill: #afffff}
|
||||
.c-160 {fill: #d70000}
|
||||
.c-161 {fill: #d7005f}
|
||||
.c-162 {fill: #d70087}
|
||||
.c-163 {fill: #d700af}
|
||||
.c-164 {fill: #d700d7}
|
||||
.c-165 {fill: #d700ff}
|
||||
.c-166 {fill: #d75f00}
|
||||
.c-167 {fill: #d75f5f}
|
||||
.c-168 {fill: #d75f87}
|
||||
.c-169 {fill: #d75faf}
|
||||
.c-170 {fill: #d75fd7}
|
||||
.c-171 {fill: #d75fff}
|
||||
.c-172 {fill: #d78700}
|
||||
.c-173 {fill: #d7875f}
|
||||
.c-174 {fill: #d78787}
|
||||
.c-175 {fill: #d787af}
|
||||
.c-176 {fill: #d787d7}
|
||||
.c-177 {fill: #d787ff}
|
||||
.c-178 {fill: #d7af00}
|
||||
.c-179 {fill: #d7af5f}
|
||||
.c-180 {fill: #d7af87}
|
||||
.c-181 {fill: #d7afaf}
|
||||
.c-182 {fill: #d7afd7}
|
||||
.c-183 {fill: #d7afff}
|
||||
.c-184 {fill: #d7d700}
|
||||
.c-185 {fill: #d7d75f}
|
||||
.c-186 {fill: #d7d787}
|
||||
.c-187 {fill: #d7d7af}
|
||||
.c-188 {fill: #d7d7d7}
|
||||
.c-189 {fill: #d7d7ff}
|
||||
.c-190 {fill: #d7ff00}
|
||||
.c-191 {fill: #d7ff5f}
|
||||
.c-192 {fill: #d7ff87}
|
||||
.c-193 {fill: #d7ffaf}
|
||||
.c-194 {fill: #d7ffd7}
|
||||
.c-195 {fill: #d7ffff}
|
||||
.c-196 {fill: #ff0000}
|
||||
.c-197 {fill: #ff005f}
|
||||
.c-198 {fill: #ff0087}
|
||||
.c-199 {fill: #ff00af}
|
||||
.c-200 {fill: #ff00d7}
|
||||
.c-201 {fill: #ff00ff}
|
||||
.c-202 {fill: #ff5f00}
|
||||
.c-203 {fill: #ff5f5f}
|
||||
.c-204 {fill: #ff5f87}
|
||||
.c-205 {fill: #ff5faf}
|
||||
.c-206 {fill: #ff5fd7}
|
||||
.c-207 {fill: #ff5fff}
|
||||
.c-208 {fill: #ff8700}
|
||||
.c-209 {fill: #ff875f}
|
||||
.c-210 {fill: #ff8787}
|
||||
.c-211 {fill: #ff87af}
|
||||
.c-212 {fill: #ff87d7}
|
||||
.c-213 {fill: #ff87ff}
|
||||
.c-214 {fill: #ffaf00}
|
||||
.c-215 {fill: #ffaf5f}
|
||||
.c-216 {fill: #ffaf87}
|
||||
.c-217 {fill: #ffafaf}
|
||||
.c-218 {fill: #ffafd7}
|
||||
.c-219 {fill: #ffafff}
|
||||
.c-220 {fill: #ffd700}
|
||||
.c-221 {fill: #ffd75f}
|
||||
.c-222 {fill: #ffd787}
|
||||
.c-223 {fill: #ffd7af}
|
||||
.c-224 {fill: #ffd7d7}
|
||||
.c-225 {fill: #ffd7ff}
|
||||
.c-226 {fill: #ffff00}
|
||||
.c-227 {fill: #ffff5f}
|
||||
.c-228 {fill: #ffff87}
|
||||
.c-229 {fill: #ffffaf}
|
||||
.c-230 {fill: #ffffd7}
|
||||
.c-231 {fill: #ffffff}
|
||||
.c-232 {fill: #080808}
|
||||
.c-233 {fill: #121212}
|
||||
.c-234 {fill: #1c1c1c}
|
||||
.c-235 {fill: #262626}
|
||||
.c-236 {fill: #303030}
|
||||
.c-237 {fill: #3a3a3a}
|
||||
.c-238 {fill: #444444}
|
||||
.c-239 {fill: #4e4e4e}
|
||||
.c-240 {fill: #585858}
|
||||
.c-241 {fill: #626262}
|
||||
.c-242 {fill: #6c6c6c}
|
||||
.c-243 {fill: #767676}
|
||||
.c-244 {fill: #808080}
|
||||
.c-245 {fill: #8a8a8a}
|
||||
.c-246 {fill: #949494}
|
||||
.c-247 {fill: #9e9e9e}
|
||||
.c-248 {fill: #a8a8a8}
|
||||
.c-249 {fill: #b2b2b2}
|
||||
.c-250 {fill: #bcbcbc}
|
||||
.c-251 {fill: #c6c6c6}
|
||||
.c-252 {fill: #d0d0d0}
|
||||
.c-253 {fill: #dadada}
|
||||
.c-254 {fill: #e4e4e4}
|
||||
.c-255 {fill: #eeeeee}
|
||||
|
||||
.br { font-weight: bold }
|
||||
.it { font-style: italic }
|
||||
.un { text-decoration: underline }
|
||||
</style>
|
||||
|
||||
<rect width="100%" height="100%" class="default-bg-fill" />
|
||||
|
||||
<svg x="0.625%" y="1.136%" class="default-text-fill">
|
||||
<g style="shape-rendering: optimizeSpeed">
|
||||
<rect x="5.625%" y="6.818%" width="0.625%" height="19.7" class="c-7" />
|
||||
<rect x="0.000%" y="95.455%" width="98.750%" height="19.7" class="c-2" />
|
||||
</g>
|
||||
|
||||
<text class="default-text-fill">
|
||||
<tspan y="0.000%">
|
||||
<tspan dy="1em" x="0.000%">c</tspan><tspan x="0.625%">h</tspan><tspan x="1.250%">a</tspan><tspan x="1.875%">o</tspan><tspan x="2.500%">s</tspan><tspan x="3.125%">:</tspan><tspan x="3.750%">~</tspan><tspan x="4.375%">$</tspan><tspan x="5.625%">#</tspan><tspan x="6.875%">B</tspan><tspan x="7.500%">u</tspan><tspan x="8.125%">i</tspan><tspan x="8.750%">l</tspan><tspan x="9.375%">d</tspan><tspan x="10.625%">a</tspan><tspan x="11.250%">n</tspan><tspan x="11.875%">d</tspan><tspan x="13.125%">a</tspan><tspan x="13.750%">p</tspan><tspan x="14.375%">p</tspan><tspan x="15.000%">l</tspan><tspan x="15.625%">y</tspan><tspan x="16.875%">C</tspan><tspan x="17.500%">h</tspan><tspan x="18.125%">a</tspan><tspan x="18.750%">o</tspan><tspan x="19.375%">s</tspan><tspan x="20.000%">E</tspan><tspan x="20.625%">n</tspan><tspan x="21.250%">g</tspan><tspan x="21.875%">i</tspan><tspan x="22.500%">n</tspan><tspan x="23.125%">e</tspan><tspan x="24.375%">C</tspan><tspan x="25.000%">R</tspan><tspan x="26.250%">t</tspan><tspan x="26.875%">o</tspan><tspan x="28.125%">u</tspan><tspan x="28.750%">n</tspan><tspan x="29.375%">l</tspan><tspan x="30.000%">e</tspan><tspan x="30.625%">a</tspan><tspan x="31.250%">s</tspan><tspan x="31.875%">h</tspan><tspan x="33.125%">C</tspan><tspan x="33.750%">h</tspan><tspan x="34.375%">a</tspan><tspan x="35.000%">o</tspan><tspan x="35.625%">s</tspan><tspan x="49.375%" class="c-2">│</tspan><tspan x="50.000%">E</tspan><tspan x="50.625%">v</tspan><tspan x="51.250%">e</tspan><tspan x="51.875%">r</tspan><tspan x="52.500%">y</tspan><tspan x="53.750%">1</tspan><tspan x="54.375%">.</tspan><tspan x="55.000%">0</tspan><tspan x="55.625%">s</tspan><tspan x="56.250%">:</tspan><tspan x="57.500%">k</tspan><tspan x="58.125%">u</tspan><tspan x="58.750%">b</tspan><tspan x="59.375%">e</tspan><tspan x="60.000%">c</tspan><tspan x="60.625%">t</tspan><tspan x="61.250%">l</tspan><tspan x="62.500%">g</tspan><tspan x="63.125%">e</tspan><tspan x="63.750%">t</tspan><tspan x="65.000%">p</tspan><tspan x="65.625%">o</tspan><tspan x="83.750%">F</tspan><tspan x="84.375%">r</tspan><tspan x="85.000%">i</tspan><tspan x="86.250%">O</tspan><tspan x="86.875%">c</tspan><tspan x="87.500%">t</tspan><tspan x="89.375%">4</tspan><tspan x="90.625%">1</tspan><tspan x="91.250%">9</tspan><tspan x="91.875%">:</tspan><tspan x="92.500%">3</tspan><tspan x="93.125%">2</tspan><tspan x="93.750%">:</tspan><tspan x="94.375%">3</tspan><tspan x="95.000%">5</tspan><tspan x="96.250%">2</tspan><tspan x="96.875%">0</tspan><tspan x="97.500%">1</tspan><tspan x="98.125%">9</tspan>
|
||||
</tspan>
|
||||
<tspan y="2.273%">
|
||||
<tspan dy="1em" x="0.000%">c</tspan><tspan x="0.625%">h</tspan><tspan x="1.250%">a</tspan><tspan x="1.875%">o</tspan><tspan x="2.500%">s</tspan><tspan x="3.125%">:</tspan><tspan x="3.750%">~</tspan><tspan x="4.375%">$</tspan><tspan x="49.375%" class="c-2">│</tspan>
|
||||
</tspan>
|
||||
<tspan y="4.545%">
|
||||
<tspan dy="1em" x="0.000%">c</tspan><tspan x="0.625%">h</tspan><tspan x="1.250%">a</tspan><tspan x="1.875%">o</tspan><tspan x="2.500%">s</tspan><tspan x="3.125%">:</tspan><tspan x="3.750%">~</tspan><tspan x="4.375%">$</tspan><tspan x="5.625%">v</tspan><tspan x="6.250%">i</tspan><tspan x="7.500%">c</tspan><tspan x="8.125%">h</tspan><tspan x="8.750%">a</tspan><tspan x="9.375%">o</tspan><tspan x="10.000%">s</tspan><tspan x="10.625%">e</tspan><tspan x="11.250%">n</tspan><tspan x="11.875%">g</tspan><tspan x="12.500%">i</tspan><tspan x="13.125%">n</tspan><tspan x="13.750%">e</tspan><tspan x="14.375%">.</tspan><tspan x="15.000%">y</tspan><tspan x="15.625%">a</tspan><tspan x="16.250%">m</tspan><tspan x="16.875%">l</tspan><tspan x="49.375%" class="c-2">│</tspan><tspan x="50.000%">N</tspan><tspan x="50.625%">A</tspan><tspan x="51.250%">M</tspan><tspan x="51.875%">E</tspan><tspan x="69.375%">R</tspan><tspan x="70.000%">E</tspan><tspan x="70.625%">A</tspan><tspan x="71.250%">D</tspan><tspan x="71.875%">Y</tspan><tspan x="74.375%">S</tspan><tspan x="75.000%">T</tspan><tspan x="75.625%">A</tspan><tspan x="76.250%">T</tspan><tspan x="76.875%">U</tspan><tspan x="77.500%">S</tspan><tspan x="80.625%">R</tspan><tspan x="81.250%">E</tspan><tspan x="81.875%">S</tspan><tspan x="82.500%">T</tspan><tspan x="83.125%">A</tspan><tspan x="83.750%">R</tspan><tspan x="84.375%">T</tspan><tspan x="85.000%">S</tspan><tspan x="87.500%">A</tspan><tspan x="88.125%">G</tspan><tspan x="88.750%">E</tspan>
|
||||
</tspan>
|
||||
<tspan y="6.818%">
|
||||
<tspan dy="1em" x="0.000%">c</tspan><tspan x="0.625%">h</tspan><tspan x="1.250%">a</tspan><tspan x="1.875%">o</tspan><tspan x="2.500%">s</tspan><tspan x="3.125%">:</tspan><tspan x="3.750%">~</tspan><tspan x="4.375%">$</tspan><tspan x="49.375%" class="c-2">│</tspan><tspan x="50.000%">h</tspan><tspan x="50.625%">e</tspan><tspan x="51.250%">l</tspan><tspan x="51.875%">l</tspan><tspan x="52.500%">o</tspan><tspan x="53.125%">-</tspan><tspan x="53.750%">d</tspan><tspan x="54.375%">e</tspan><tspan x="55.000%">p</tspan><tspan x="55.625%">l</tspan><tspan x="56.250%">o</tspan><tspan x="56.875%">y</tspan><tspan x="57.500%">-</tspan><tspan x="58.125%">d</tspan><tspan x="58.750%">d</tspan><tspan x="59.375%">5</tspan><tspan x="60.000%">9</tspan><tspan x="60.625%">b</tspan><tspan x="61.250%">8</tspan><tspan x="61.875%">9</tspan><tspan x="62.500%">5</tspan><tspan x="63.125%">6</tspan><tspan x="63.750%">-</tspan><tspan x="64.375%">h</tspan><tspan x="65.000%">x</tspan><tspan x="65.625%">c</tspan><tspan x="66.250%">j</tspan><tspan x="66.875%">v</tspan><tspan x="69.375%">1</tspan><tspan x="70.000%">/</tspan><tspan x="70.625%">1</tspan><tspan x="74.375%">R</tspan><tspan x="75.000%">u</tspan><tspan x="75.625%">n</tspan><tspan x="76.250%">n</tspan><tspan x="76.875%">i</tspan><tspan x="77.500%">n</tspan><tspan x="78.125%">g</tspan><tspan x="80.625%">0</tspan><tspan x="87.500%">1</tspan><tspan x="88.125%">9</tspan><tspan x="88.750%">m</tspan>
|
||||
</tspan>
|
||||
<tspan y="9.091%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan>
|
||||
</tspan>
|
||||
<tspan y="11.364%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan>
|
||||
</tspan>
|
||||
<tspan y="13.636%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan>
|
||||
</tspan>
|
||||
<tspan y="15.909%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan>
|
||||
</tspan>
|
||||
<tspan y="18.182%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan>
|
||||
</tspan>
|
||||
<tspan y="20.455%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan>
|
||||
</tspan>
|
||||
<tspan y="22.727%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan>
|
||||
</tspan>
|
||||
<tspan y="25.000%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan>
|
||||
</tspan>
|
||||
<tspan y="27.273%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan>
|
||||
</tspan>
|
||||
<tspan y="29.545%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan>
|
||||
</tspan>
|
||||
<tspan y="31.818%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan>
|
||||
</tspan>
|
||||
<tspan y="34.091%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan>
|
||||
</tspan>
|
||||
<tspan y="36.364%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan>
|
||||
</tspan>
|
||||
<tspan y="38.636%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan>
|
||||
</tspan>
|
||||
<tspan y="40.909%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan>
|
||||
</tspan>
|
||||
<tspan y="43.182%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan>
|
||||
</tspan>
|
||||
<tspan y="45.455%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan>
|
||||
</tspan>
|
||||
<tspan y="47.727%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">├</tspan><tspan x="50.000%">─</tspan><tspan x="50.625%">─</tspan><tspan x="51.250%">─</tspan><tspan x="51.875%">─</tspan><tspan x="52.500%">─</tspan><tspan x="53.125%">─</tspan><tspan x="53.750%">─</tspan><tspan x="54.375%">─</tspan><tspan x="55.000%">─</tspan><tspan x="55.625%">─</tspan><tspan x="56.250%">─</tspan><tspan x="56.875%">─</tspan><tspan x="57.500%">─</tspan><tspan x="58.125%">─</tspan><tspan x="58.750%">─</tspan><tspan x="59.375%">─</tspan><tspan x="60.000%">─</tspan><tspan x="60.625%">─</tspan><tspan x="61.250%">─</tspan><tspan x="61.875%">─</tspan><tspan x="62.500%">─</tspan><tspan x="63.125%">─</tspan><tspan x="63.750%">─</tspan><tspan x="64.375%">─</tspan><tspan x="65.000%">─</tspan><tspan x="65.625%">─</tspan><tspan x="66.250%">─</tspan><tspan x="66.875%">─</tspan><tspan x="67.500%">─</tspan><tspan x="68.125%">─</tspan><tspan x="68.750%">─</tspan><tspan x="69.375%">─</tspan><tspan x="70.000%">─</tspan><tspan x="70.625%">─</tspan><tspan x="71.250%">─</tspan><tspan x="71.875%">─</tspan><tspan x="72.500%">─</tspan><tspan x="73.125%">─</tspan><tspan x="73.750%">─</tspan><tspan x="74.375%">─</tspan><tspan x="75.000%">─</tspan><tspan x="75.625%">─</tspan><tspan x="76.250%">─</tspan><tspan x="76.875%">─</tspan><tspan x="77.500%">─</tspan><tspan x="78.125%">─</tspan><tspan x="78.750%">─</tspan><tspan x="79.375%">─</tspan><tspan x="80.000%">─</tspan><tspan x="80.625%">─</tspan><tspan x="81.250%">─</tspan><tspan x="81.875%">─</tspan><tspan x="82.500%">─</tspan><tspan x="83.125%">─</tspan><tspan x="83.750%">─</tspan><tspan x="84.375%">─</tspan><tspan x="85.000%">─</tspan><tspan x="85.625%">─</tspan><tspan x="86.250%">─</tspan><tspan x="86.875%">─</tspan><tspan x="87.500%">─</tspan><tspan x="88.125%">─</tspan><tspan x="88.750%">─</tspan><tspan x="89.375%">─</tspan><tspan x="90.000%">─</tspan><tspan x="90.625%">─</tspan><tspan x="91.250%">─</tspan><tspan x="91.875%">─</tspan><tspan x="92.500%">─</tspan><tspan x="93.125%">─</tspan><tspan x="93.750%">─</tspan><tspan x="94.375%">─</tspan><tspan x="95.000%">─</tspan><tspan x="95.625%">─</tspan><tspan x="96.250%">─</tspan><tspan x="96.875%">─</tspan><tspan x="97.500%">─</tspan><tspan x="98.125%">─</tspan>
|
||||
</tspan>
|
||||
<tspan y="50.000%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan><tspan x="50.000%" class="br c-10">H</tspan><tspan x="50.625%" class="br c-10">e</tspan><tspan x="51.250%" class="br c-10">l</tspan><tspan x="51.875%" class="br c-10">l</tspan><tspan x="52.500%" class="br c-10">o</tspan><tspan x="53.750%" class="br c-10">W</tspan><tspan x="54.375%" class="br c-10">o</tspan><tspan x="55.000%" class="br c-10">r</tspan><tspan x="55.625%" class="br c-10">l</tspan><tspan x="56.250%" class="br c-10">d</tspan><tspan x="57.500%" class="br c-10">i</tspan><tspan x="58.125%" class="br c-10">s</tspan><tspan x="59.375%" class="br c-10">o</tspan><tspan x="60.000%" class="br c-10">n</tspan><tspan x="60.625%" class="br c-10">l</tspan><tspan x="61.250%" class="br c-10">i</tspan><tspan x="61.875%" class="br c-10">n</tspan><tspan x="62.500%" class="br c-10">e</tspan><tspan x="63.750%">H</tspan><tspan x="64.375%">T</tspan><tspan x="65.000%">T</tspan><tspan x="65.625%">P</tspan><tspan x="66.250%">/</tspan><tspan x="66.875%">2</tspan><tspan x="68.125%">2</tspan><tspan x="68.750%">0</tspan><tspan x="69.375%">0</tspan><tspan x="70.625%">O</tspan><tspan x="71.250%">K</tspan>
|
||||
</tspan>
|
||||
<tspan y="52.273%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan><tspan x="50.000%" class="br c-10">H</tspan><tspan x="50.625%" class="br c-10">e</tspan><tspan x="51.250%" class="br c-10">l</tspan><tspan x="51.875%" class="br c-10">l</tspan><tspan x="52.500%" class="br c-10">o</tspan><tspan x="53.750%" class="br c-10">W</tspan><tspan x="54.375%" class="br c-10">o</tspan><tspan x="55.000%" class="br c-10">r</tspan><tspan x="55.625%" class="br c-10">l</tspan><tspan x="56.250%" class="br c-10">d</tspan><tspan x="57.500%" class="br c-10">i</tspan><tspan x="58.125%" class="br c-10">s</tspan><tspan x="59.375%" class="br c-10">o</tspan><tspan x="60.000%" class="br c-10">n</tspan><tspan x="60.625%" class="br c-10">l</tspan><tspan x="61.250%" class="br c-10">i</tspan><tspan x="61.875%" class="br c-10">n</tspan><tspan x="62.500%" class="br c-10">e</tspan><tspan x="63.750%">H</tspan><tspan x="64.375%">T</tspan><tspan x="65.000%">T</tspan><tspan x="65.625%">P</tspan><tspan x="66.250%">/</tspan><tspan x="66.875%">2</tspan><tspan x="68.125%">2</tspan><tspan x="68.750%">0</tspan><tspan x="69.375%">0</tspan><tspan x="70.625%">O</tspan><tspan x="71.250%">K</tspan>
|
||||
</tspan>
|
||||
<tspan y="54.545%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan><tspan x="50.000%" class="br c-10">H</tspan><tspan x="50.625%" class="br c-10">e</tspan><tspan x="51.250%" class="br c-10">l</tspan><tspan x="51.875%" class="br c-10">l</tspan><tspan x="52.500%" class="br c-10">o</tspan><tspan x="53.750%" class="br c-10">W</tspan><tspan x="54.375%" class="br c-10">o</tspan><tspan x="55.000%" class="br c-10">r</tspan><tspan x="55.625%" class="br c-10">l</tspan><tspan x="56.250%" class="br c-10">d</tspan><tspan x="57.500%" class="br c-10">i</tspan><tspan x="58.125%" class="br c-10">s</tspan><tspan x="59.375%" class="br c-10">o</tspan><tspan x="60.000%" class="br c-10">n</tspan><tspan x="60.625%" class="br c-10">l</tspan><tspan x="61.250%" class="br c-10">i</tspan><tspan x="61.875%" class="br c-10">n</tspan><tspan x="62.500%" class="br c-10">e</tspan><tspan x="63.750%">H</tspan><tspan x="64.375%">T</tspan><tspan x="65.000%">T</tspan><tspan x="65.625%">P</tspan><tspan x="66.250%">/</tspan><tspan x="66.875%">2</tspan><tspan x="68.125%">2</tspan><tspan x="68.750%">0</tspan><tspan x="69.375%">0</tspan><tspan x="70.625%">O</tspan><tspan x="71.250%">K</tspan>
|
||||
</tspan>
|
||||
<tspan y="56.818%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan><tspan x="50.000%" class="br c-10">H</tspan><tspan x="50.625%" class="br c-10">e</tspan><tspan x="51.250%" class="br c-10">l</tspan><tspan x="51.875%" class="br c-10">l</tspan><tspan x="52.500%" class="br c-10">o</tspan><tspan x="53.750%" class="br c-10">W</tspan><tspan x="54.375%" class="br c-10">o</tspan><tspan x="55.000%" class="br c-10">r</tspan><tspan x="55.625%" class="br c-10">l</tspan><tspan x="56.250%" class="br c-10">d</tspan><tspan x="57.500%" class="br c-10">i</tspan><tspan x="58.125%" class="br c-10">s</tspan><tspan x="59.375%" class="br c-10">o</tspan><tspan x="60.000%" class="br c-10">n</tspan><tspan x="60.625%" class="br c-10">l</tspan><tspan x="61.250%" class="br c-10">i</tspan><tspan x="61.875%" class="br c-10">n</tspan><tspan x="62.500%" class="br c-10">e</tspan><tspan x="63.750%">H</tspan><tspan x="64.375%">T</tspan><tspan x="65.000%">T</tspan><tspan x="65.625%">P</tspan><tspan x="66.250%">/</tspan><tspan x="66.875%">2</tspan><tspan x="68.125%">2</tspan><tspan x="68.750%">0</tspan><tspan x="69.375%">0</tspan><tspan x="70.625%">O</tspan><tspan x="71.250%">K</tspan>
|
||||
</tspan>
|
||||
<tspan y="59.091%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan><tspan x="50.000%" class="br c-10">H</tspan><tspan x="50.625%" class="br c-10">e</tspan><tspan x="51.250%" class="br c-10">l</tspan><tspan x="51.875%" class="br c-10">l</tspan><tspan x="52.500%" class="br c-10">o</tspan><tspan x="53.750%" class="br c-10">W</tspan><tspan x="54.375%" class="br c-10">o</tspan><tspan x="55.000%" class="br c-10">r</tspan><tspan x="55.625%" class="br c-10">l</tspan><tspan x="56.250%" class="br c-10">d</tspan><tspan x="57.500%" class="br c-10">i</tspan><tspan x="58.125%" class="br c-10">s</tspan><tspan x="59.375%" class="br c-10">o</tspan><tspan x="60.000%" class="br c-10">n</tspan><tspan x="60.625%" class="br c-10">l</tspan><tspan x="61.250%" class="br c-10">i</tspan><tspan x="61.875%" class="br c-10">n</tspan><tspan x="62.500%" class="br c-10">e</tspan><tspan x="63.750%">H</tspan><tspan x="64.375%">T</tspan><tspan x="65.000%">T</tspan><tspan x="65.625%">P</tspan><tspan x="66.250%">/</tspan><tspan x="66.875%">2</tspan><tspan x="68.125%">2</tspan><tspan x="68.750%">0</tspan><tspan x="69.375%">0</tspan><tspan x="70.625%">O</tspan><tspan x="71.250%">K</tspan>
|
||||
</tspan>
|
||||
<tspan y="61.364%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan><tspan x="50.000%" class="br c-10">H</tspan><tspan x="50.625%" class="br c-10">e</tspan><tspan x="51.250%" class="br c-10">l</tspan><tspan x="51.875%" class="br c-10">l</tspan><tspan x="52.500%" class="br c-10">o</tspan><tspan x="53.750%" class="br c-10">W</tspan><tspan x="54.375%" class="br c-10">o</tspan><tspan x="55.000%" class="br c-10">r</tspan><tspan x="55.625%" class="br c-10">l</tspan><tspan x="56.250%" class="br c-10">d</tspan><tspan x="57.500%" class="br c-10">i</tspan><tspan x="58.125%" class="br c-10">s</tspan><tspan x="59.375%" class="br c-10">o</tspan><tspan x="60.000%" class="br c-10">n</tspan><tspan x="60.625%" class="br c-10">l</tspan><tspan x="61.250%" class="br c-10">i</tspan><tspan x="61.875%" class="br c-10">n</tspan><tspan x="62.500%" class="br c-10">e</tspan><tspan x="63.750%">H</tspan><tspan x="64.375%">T</tspan><tspan x="65.000%">T</tspan><tspan x="65.625%">P</tspan><tspan x="66.250%">/</tspan><tspan x="66.875%">2</tspan><tspan x="68.125%">2</tspan><tspan x="68.750%">0</tspan><tspan x="69.375%">0</tspan><tspan x="70.625%">O</tspan><tspan x="71.250%">K</tspan>
|
||||
</tspan>
|
||||
<tspan y="63.636%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan><tspan x="50.000%" class="br c-10">H</tspan><tspan x="50.625%" class="br c-10">e</tspan><tspan x="51.250%" class="br c-10">l</tspan><tspan x="51.875%" class="br c-10">l</tspan><tspan x="52.500%" class="br c-10">o</tspan><tspan x="53.750%" class="br c-10">W</tspan><tspan x="54.375%" class="br c-10">o</tspan><tspan x="55.000%" class="br c-10">r</tspan><tspan x="55.625%" class="br c-10">l</tspan><tspan x="56.250%" class="br c-10">d</tspan><tspan x="57.500%" class="br c-10">i</tspan><tspan x="58.125%" class="br c-10">s</tspan><tspan x="59.375%" class="br c-10">o</tspan><tspan x="60.000%" class="br c-10">n</tspan><tspan x="60.625%" class="br c-10">l</tspan><tspan x="61.250%" class="br c-10">i</tspan><tspan x="61.875%" class="br c-10">n</tspan><tspan x="62.500%" class="br c-10">e</tspan><tspan x="63.750%">H</tspan><tspan x="64.375%">T</tspan><tspan x="65.000%">T</tspan><tspan x="65.625%">P</tspan><tspan x="66.250%">/</tspan><tspan x="66.875%">2</tspan><tspan x="68.125%">2</tspan><tspan x="68.750%">0</tspan><tspan x="69.375%">0</tspan><tspan x="70.625%">O</tspan><tspan x="71.250%">K</tspan>
|
||||
</tspan>
|
||||
<tspan y="65.909%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan><tspan x="50.000%" class="br c-10">H</tspan><tspan x="50.625%" class="br c-10">e</tspan><tspan x="51.250%" class="br c-10">l</tspan><tspan x="51.875%" class="br c-10">l</tspan><tspan x="52.500%" class="br c-10">o</tspan><tspan x="53.750%" class="br c-10">W</tspan><tspan x="54.375%" class="br c-10">o</tspan><tspan x="55.000%" class="br c-10">r</tspan><tspan x="55.625%" class="br c-10">l</tspan><tspan x="56.250%" class="br c-10">d</tspan><tspan x="57.500%" class="br c-10">i</tspan><tspan x="58.125%" class="br c-10">s</tspan><tspan x="59.375%" class="br c-10">o</tspan><tspan x="60.000%" class="br c-10">n</tspan><tspan x="60.625%" class="br c-10">l</tspan><tspan x="61.250%" class="br c-10">i</tspan><tspan x="61.875%" class="br c-10">n</tspan><tspan x="62.500%" class="br c-10">e</tspan><tspan x="63.750%">H</tspan><tspan x="64.375%">T</tspan><tspan x="65.000%">T</tspan><tspan x="65.625%">P</tspan><tspan x="66.250%">/</tspan><tspan x="66.875%">2</tspan><tspan x="68.125%">2</tspan><tspan x="68.750%">0</tspan><tspan x="69.375%">0</tspan><tspan x="70.625%">O</tspan><tspan x="71.250%">K</tspan>
|
||||
</tspan>
|
||||
<tspan y="68.182%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan><tspan x="50.000%" class="br c-10">H</tspan><tspan x="50.625%" class="br c-10">e</tspan><tspan x="51.250%" class="br c-10">l</tspan><tspan x="51.875%" class="br c-10">l</tspan><tspan x="52.500%" class="br c-10">o</tspan><tspan x="53.750%" class="br c-10">W</tspan><tspan x="54.375%" class="br c-10">o</tspan><tspan x="55.000%" class="br c-10">r</tspan><tspan x="55.625%" class="br c-10">l</tspan><tspan x="56.250%" class="br c-10">d</tspan><tspan x="57.500%" class="br c-10">i</tspan><tspan x="58.125%" class="br c-10">s</tspan><tspan x="59.375%" class="br c-10">o</tspan><tspan x="60.000%" class="br c-10">n</tspan><tspan x="60.625%" class="br c-10">l</tspan><tspan x="61.250%" class="br c-10">i</tspan><tspan x="61.875%" class="br c-10">n</tspan><tspan x="62.500%" class="br c-10">e</tspan><tspan x="63.750%">H</tspan><tspan x="64.375%">T</tspan><tspan x="65.000%">T</tspan><tspan x="65.625%">P</tspan><tspan x="66.250%">/</tspan><tspan x="66.875%">2</tspan><tspan x="68.125%">2</tspan><tspan x="68.750%">0</tspan><tspan x="69.375%">0</tspan><tspan x="70.625%">O</tspan><tspan x="71.250%">K</tspan>
|
||||
</tspan>
|
||||
<tspan y="70.455%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan><tspan x="50.000%" class="br c-10">H</tspan><tspan x="50.625%" class="br c-10">e</tspan><tspan x="51.250%" class="br c-10">l</tspan><tspan x="51.875%" class="br c-10">l</tspan><tspan x="52.500%" class="br c-10">o</tspan><tspan x="53.750%" class="br c-10">W</tspan><tspan x="54.375%" class="br c-10">o</tspan><tspan x="55.000%" class="br c-10">r</tspan><tspan x="55.625%" class="br c-10">l</tspan><tspan x="56.250%" class="br c-10">d</tspan><tspan x="57.500%" class="br c-10">i</tspan><tspan x="58.125%" class="br c-10">s</tspan><tspan x="59.375%" class="br c-10">o</tspan><tspan x="60.000%" class="br c-10">n</tspan><tspan x="60.625%" class="br c-10">l</tspan><tspan x="61.250%" class="br c-10">i</tspan><tspan x="61.875%" class="br c-10">n</tspan><tspan x="62.500%" class="br c-10">e</tspan><tspan x="63.750%">H</tspan><tspan x="64.375%">T</tspan><tspan x="65.000%">T</tspan><tspan x="65.625%">P</tspan><tspan x="66.250%">/</tspan><tspan x="66.875%">2</tspan><tspan x="68.125%">2</tspan><tspan x="68.750%">0</tspan><tspan x="69.375%">0</tspan><tspan x="70.625%">O</tspan><tspan x="71.250%">K</tspan>
|
||||
</tspan>
|
||||
<tspan y="72.727%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan><tspan x="50.000%" class="br c-10">H</tspan><tspan x="50.625%" class="br c-10">e</tspan><tspan x="51.250%" class="br c-10">l</tspan><tspan x="51.875%" class="br c-10">l</tspan><tspan x="52.500%" class="br c-10">o</tspan><tspan x="53.750%" class="br c-10">W</tspan><tspan x="54.375%" class="br c-10">o</tspan><tspan x="55.000%" class="br c-10">r</tspan><tspan x="55.625%" class="br c-10">l</tspan><tspan x="56.250%" class="br c-10">d</tspan><tspan x="57.500%" class="br c-10">i</tspan><tspan x="58.125%" class="br c-10">s</tspan><tspan x="59.375%" class="br c-10">o</tspan><tspan x="60.000%" class="br c-10">n</tspan><tspan x="60.625%" class="br c-10">l</tspan><tspan x="61.250%" class="br c-10">i</tspan><tspan x="61.875%" class="br c-10">n</tspan><tspan x="62.500%" class="br c-10">e</tspan><tspan x="63.750%">H</tspan><tspan x="64.375%">T</tspan><tspan x="65.000%">T</tspan><tspan x="65.625%">P</tspan><tspan x="66.250%">/</tspan><tspan x="66.875%">2</tspan><tspan x="68.125%">2</tspan><tspan x="68.750%">0</tspan><tspan x="69.375%">0</tspan><tspan x="70.625%">O</tspan><tspan x="71.250%">K</tspan>
|
||||
</tspan>
|
||||
<tspan y="75.000%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan><tspan x="50.000%" class="br c-10">H</tspan><tspan x="50.625%" class="br c-10">e</tspan><tspan x="51.250%" class="br c-10">l</tspan><tspan x="51.875%" class="br c-10">l</tspan><tspan x="52.500%" class="br c-10">o</tspan><tspan x="53.750%" class="br c-10">W</tspan><tspan x="54.375%" class="br c-10">o</tspan><tspan x="55.000%" class="br c-10">r</tspan><tspan x="55.625%" class="br c-10">l</tspan><tspan x="56.250%" class="br c-10">d</tspan><tspan x="57.500%" class="br c-10">i</tspan><tspan x="58.125%" class="br c-10">s</tspan><tspan x="59.375%" class="br c-10">o</tspan><tspan x="60.000%" class="br c-10">n</tspan><tspan x="60.625%" class="br c-10">l</tspan><tspan x="61.250%" class="br c-10">i</tspan><tspan x="61.875%" class="br c-10">n</tspan><tspan x="62.500%" class="br c-10">e</tspan><tspan x="63.750%">H</tspan><tspan x="64.375%">T</tspan><tspan x="65.000%">T</tspan><tspan x="65.625%">P</tspan><tspan x="66.250%">/</tspan><tspan x="66.875%">2</tspan><tspan x="68.125%">2</tspan><tspan x="68.750%">0</tspan><tspan x="69.375%">0</tspan><tspan x="70.625%">O</tspan><tspan x="71.250%">K</tspan>
|
||||
</tspan>
|
||||
<tspan y="77.273%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan><tspan x="50.000%" class="br c-10">H</tspan><tspan x="50.625%" class="br c-10">e</tspan><tspan x="51.250%" class="br c-10">l</tspan><tspan x="51.875%" class="br c-10">l</tspan><tspan x="52.500%" class="br c-10">o</tspan><tspan x="53.750%" class="br c-10">W</tspan><tspan x="54.375%" class="br c-10">o</tspan><tspan x="55.000%" class="br c-10">r</tspan><tspan x="55.625%" class="br c-10">l</tspan><tspan x="56.250%" class="br c-10">d</tspan><tspan x="57.500%" class="br c-10">i</tspan><tspan x="58.125%" class="br c-10">s</tspan><tspan x="59.375%" class="br c-10">o</tspan><tspan x="60.000%" class="br c-10">n</tspan><tspan x="60.625%" class="br c-10">l</tspan><tspan x="61.250%" class="br c-10">i</tspan><tspan x="61.875%" class="br c-10">n</tspan><tspan x="62.500%" class="br c-10">e</tspan><tspan x="63.750%">H</tspan><tspan x="64.375%">T</tspan><tspan x="65.000%">T</tspan><tspan x="65.625%">P</tspan><tspan x="66.250%">/</tspan><tspan x="66.875%">2</tspan><tspan x="68.125%">2</tspan><tspan x="68.750%">0</tspan><tspan x="69.375%">0</tspan><tspan x="70.625%">O</tspan><tspan x="71.250%">K</tspan>
|
||||
</tspan>
|
||||
<tspan y="79.545%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan><tspan x="50.000%" class="br c-10">H</tspan><tspan x="50.625%" class="br c-10">e</tspan><tspan x="51.250%" class="br c-10">l</tspan><tspan x="51.875%" class="br c-10">l</tspan><tspan x="52.500%" class="br c-10">o</tspan><tspan x="53.750%" class="br c-10">W</tspan><tspan x="54.375%" class="br c-10">o</tspan><tspan x="55.000%" class="br c-10">r</tspan><tspan x="55.625%" class="br c-10">l</tspan><tspan x="56.250%" class="br c-10">d</tspan><tspan x="57.500%" class="br c-10">i</tspan><tspan x="58.125%" class="br c-10">s</tspan><tspan x="59.375%" class="br c-10">o</tspan><tspan x="60.000%" class="br c-10">n</tspan><tspan x="60.625%" class="br c-10">l</tspan><tspan x="61.250%" class="br c-10">i</tspan><tspan x="61.875%" class="br c-10">n</tspan><tspan x="62.500%" class="br c-10">e</tspan><tspan x="63.750%">H</tspan><tspan x="64.375%">T</tspan><tspan x="65.000%">T</tspan><tspan x="65.625%">P</tspan><tspan x="66.250%">/</tspan><tspan x="66.875%">2</tspan><tspan x="68.125%">2</tspan><tspan x="68.750%">0</tspan><tspan x="69.375%">0</tspan><tspan x="70.625%">O</tspan><tspan x="71.250%">K</tspan>
|
||||
</tspan>
|
||||
<tspan y="81.818%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan><tspan x="50.000%" class="br c-10">H</tspan><tspan x="50.625%" class="br c-10">e</tspan><tspan x="51.250%" class="br c-10">l</tspan><tspan x="51.875%" class="br c-10">l</tspan><tspan x="52.500%" class="br c-10">o</tspan><tspan x="53.750%" class="br c-10">W</tspan><tspan x="54.375%" class="br c-10">o</tspan><tspan x="55.000%" class="br c-10">r</tspan><tspan x="55.625%" class="br c-10">l</tspan><tspan x="56.250%" class="br c-10">d</tspan><tspan x="57.500%" class="br c-10">i</tspan><tspan x="58.125%" class="br c-10">s</tspan><tspan x="59.375%" class="br c-10">o</tspan><tspan x="60.000%" class="br c-10">n</tspan><tspan x="60.625%" class="br c-10">l</tspan><tspan x="61.250%" class="br c-10">i</tspan><tspan x="61.875%" class="br c-10">n</tspan><tspan x="62.500%" class="br c-10">e</tspan><tspan x="63.750%">H</tspan><tspan x="64.375%">T</tspan><tspan x="65.000%">T</tspan><tspan x="65.625%">P</tspan><tspan x="66.250%">/</tspan><tspan x="66.875%">2</tspan><tspan x="68.125%">2</tspan><tspan x="68.750%">0</tspan><tspan x="69.375%">0</tspan><tspan x="70.625%">O</tspan><tspan x="71.250%">K</tspan>
|
||||
</tspan>
|
||||
<tspan y="84.091%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan><tspan x="50.000%" class="br c-10">H</tspan><tspan x="50.625%" class="br c-10">e</tspan><tspan x="51.250%" class="br c-10">l</tspan><tspan x="51.875%" class="br c-10">l</tspan><tspan x="52.500%" class="br c-10">o</tspan><tspan x="53.750%" class="br c-10">W</tspan><tspan x="54.375%" class="br c-10">o</tspan><tspan x="55.000%" class="br c-10">r</tspan><tspan x="55.625%" class="br c-10">l</tspan><tspan x="56.250%" class="br c-10">d</tspan><tspan x="57.500%" class="br c-10">i</tspan><tspan x="58.125%" class="br c-10">s</tspan><tspan x="59.375%" class="br c-10">o</tspan><tspan x="60.000%" class="br c-10">n</tspan><tspan x="60.625%" class="br c-10">l</tspan><tspan x="61.250%" class="br c-10">i</tspan><tspan x="61.875%" class="br c-10">n</tspan><tspan x="62.500%" class="br c-10">e</tspan><tspan x="63.750%">H</tspan><tspan x="64.375%">T</tspan><tspan x="65.000%">T</tspan><tspan x="65.625%">P</tspan><tspan x="66.250%">/</tspan><tspan x="66.875%">2</tspan><tspan x="68.125%">2</tspan><tspan x="68.750%">0</tspan><tspan x="69.375%">0</tspan><tspan x="70.625%">O</tspan><tspan x="71.250%">K</tspan>
|
||||
</tspan>
|
||||
<tspan y="86.364%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan><tspan x="50.000%" class="br c-10">H</tspan><tspan x="50.625%" class="br c-10">e</tspan><tspan x="51.250%" class="br c-10">l</tspan><tspan x="51.875%" class="br c-10">l</tspan><tspan x="52.500%" class="br c-10">o</tspan><tspan x="53.750%" class="br c-10">W</tspan><tspan x="54.375%" class="br c-10">o</tspan><tspan x="55.000%" class="br c-10">r</tspan><tspan x="55.625%" class="br c-10">l</tspan><tspan x="56.250%" class="br c-10">d</tspan><tspan x="57.500%" class="br c-10">i</tspan><tspan x="58.125%" class="br c-10">s</tspan><tspan x="59.375%" class="br c-10">o</tspan><tspan x="60.000%" class="br c-10">n</tspan><tspan x="60.625%" class="br c-10">l</tspan><tspan x="61.250%" class="br c-10">i</tspan><tspan x="61.875%" class="br c-10">n</tspan><tspan x="62.500%" class="br c-10">e</tspan><tspan x="63.750%">H</tspan><tspan x="64.375%">T</tspan><tspan x="65.000%">T</tspan><tspan x="65.625%">P</tspan><tspan x="66.250%">/</tspan><tspan x="66.875%">2</tspan><tspan x="68.125%">2</tspan><tspan x="68.750%">0</tspan><tspan x="69.375%">0</tspan><tspan x="70.625%">O</tspan><tspan x="71.250%">K</tspan>
|
||||
</tspan>
|
||||
<tspan y="88.636%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan><tspan x="50.000%" class="br c-10">H</tspan><tspan x="50.625%" class="br c-10">e</tspan><tspan x="51.250%" class="br c-10">l</tspan><tspan x="51.875%" class="br c-10">l</tspan><tspan x="52.500%" class="br c-10">o</tspan><tspan x="53.750%" class="br c-10">W</tspan><tspan x="54.375%" class="br c-10">o</tspan><tspan x="55.000%" class="br c-10">r</tspan><tspan x="55.625%" class="br c-10">l</tspan><tspan x="56.250%" class="br c-10">d</tspan><tspan x="57.500%" class="br c-10">i</tspan><tspan x="58.125%" class="br c-10">s</tspan><tspan x="59.375%" class="br c-10">o</tspan><tspan x="60.000%" class="br c-10">n</tspan><tspan x="60.625%" class="br c-10">l</tspan><tspan x="61.250%" class="br c-10">i</tspan><tspan x="61.875%" class="br c-10">n</tspan><tspan x="62.500%" class="br c-10">e</tspan><tspan x="63.750%">H</tspan><tspan x="64.375%">T</tspan><tspan x="65.000%">T</tspan><tspan x="65.625%">P</tspan><tspan x="66.250%">/</tspan><tspan x="66.875%">2</tspan><tspan x="68.125%">2</tspan><tspan x="68.750%">0</tspan><tspan x="69.375%">0</tspan><tspan x="70.625%">O</tspan><tspan x="71.250%">K</tspan>
|
||||
</tspan>
|
||||
<tspan y="90.909%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan><tspan x="50.000%" class="br c-10">H</tspan><tspan x="50.625%" class="br c-10">e</tspan><tspan x="51.250%" class="br c-10">l</tspan><tspan x="51.875%" class="br c-10">l</tspan><tspan x="52.500%" class="br c-10">o</tspan><tspan x="53.750%" class="br c-10">W</tspan><tspan x="54.375%" class="br c-10">o</tspan><tspan x="55.000%" class="br c-10">r</tspan><tspan x="55.625%" class="br c-10">l</tspan><tspan x="56.250%" class="br c-10">d</tspan><tspan x="57.500%" class="br c-10">i</tspan><tspan x="58.125%" class="br c-10">s</tspan><tspan x="59.375%" class="br c-10">o</tspan><tspan x="60.000%" class="br c-10">n</tspan><tspan x="60.625%" class="br c-10">l</tspan><tspan x="61.250%" class="br c-10">i</tspan><tspan x="61.875%" class="br c-10">n</tspan><tspan x="62.500%" class="br c-10">e</tspan><tspan x="63.750%">H</tspan><tspan x="64.375%">T</tspan><tspan x="65.000%">T</tspan><tspan x="65.625%">P</tspan><tspan x="66.250%">/</tspan><tspan x="66.875%">2</tspan><tspan x="68.125%">2</tspan><tspan x="68.750%">0</tspan><tspan x="69.375%">0</tspan><tspan x="70.625%">O</tspan><tspan x="71.250%">K</tspan>
|
||||
</tspan>
|
||||
<tspan y="93.182%">
|
||||
<tspan dy="1em" x="49.375%" class="c-2">│</tspan>
|
||||
</tspan>
|
||||
<tspan y="95.455%">
|
||||
<tspan dy="1em" x="0.000%" class="c-0">[</tspan><tspan x="0.625%" class="c-0">d</tspan><tspan x="1.250%" class="c-0">e</tspan><tspan x="1.875%" class="c-0">m</tspan><tspan x="2.500%" class="c-0">o</tspan><tspan x="3.125%" class="c-0">]</tspan><tspan x="4.375%" class="c-0">0</tspan><tspan x="5.000%" class="c-0">:</tspan><tspan x="5.625%" class="c-0">s</tspan><tspan x="6.250%" class="c-0">s</tspan><tspan x="6.875%" class="c-0">h</tspan><tspan x="7.500%" class="c-0">*</tspan><tspan x="75.625%" class="c-0">"</tspan><tspan x="76.250%" class="c-0">r</tspan><tspan x="76.875%" class="c-0">a</tspan><tspan x="77.500%" class="c-0">h</tspan><tspan x="78.125%" class="c-0">u</tspan><tspan x="78.750%" class="c-0">l</tspan><tspan x="79.375%" class="c-0">-</tspan><tspan x="80.000%" class="c-0">T</tspan><tspan x="80.625%" class="c-0">h</tspan><tspan x="81.250%" class="c-0">i</tspan><tspan x="81.875%" class="c-0">n</tspan><tspan x="82.500%" class="c-0">k</tspan><tspan x="83.125%" class="c-0">P</tspan><tspan x="83.750%" class="c-0">a</tspan><tspan x="84.375%" class="c-0">d</tspan><tspan x="85.000%" class="c-0">-</tspan><tspan x="85.625%" class="c-0">E</tspan><tspan x="86.250%" class="c-0">4</tspan><tspan x="86.875%" class="c-0">9</tspan><tspan x="87.500%" class="c-0">0</tspan><tspan x="88.125%" class="c-0">"</tspan><tspan x="89.375%" class="c-0">0</tspan><tspan x="90.000%" class="c-0">1</tspan><tspan x="90.625%" class="c-0">:</tspan><tspan x="91.250%" class="c-0">0</tspan><tspan x="91.875%" class="c-0">2</tspan><tspan x="93.125%" class="c-0">0</tspan><tspan x="93.750%" class="c-0">5</tspan><tspan x="94.375%" class="c-0">-</tspan><tspan x="95.000%" class="c-0">O</tspan><tspan x="95.625%" class="c-0">c</tspan><tspan x="96.250%" class="c-0">t</tspan><tspan x="96.875%" class="c-0">-</tspan><tspan x="97.500%" class="c-0">1</tspan><tspan x="98.125%" class="c-0">9</tspan>
|
||||
</tspan>
|
||||
</text>
|
||||
|
||||
<g transform="translate(-50 -50)">
|
||||
<svg x="50%" y="50%" width="100" height="100">
|
||||
<svg version="1.1" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 866.0254037844387 866.0254037844387">
|
||||
<defs>
|
||||
<mask id="small-triangle-mask">
|
||||
<rect width="100%" height="100%" fill="white"/>
|
||||
<polygon points="508.01270189221935 433.01270189221935, 208.0127018922194 259.8076211353316, 208.01270189221927 606.217782649107" fill="black"></polygon>
|
||||
</mask>
|
||||
</defs>
|
||||
<polygon points="808.0127018922194 433.01270189221935, 58.01270189221947 -1.1368683772161603e-13, 58.01270189221913 866.0254037844386" mask="url(#small-triangle-mask)" fill="white"></polygon>
|
||||
<polyline points="481.2177826491071 333.0127018922194, 134.80762113533166 533.0127018922194" stroke="white" stroke-width="90"></polyline>
|
||||
</svg>
|
||||
</svg>
|
||||
</g>
|
||||
</svg>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 41 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 47 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 124 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 29 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 55 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 96 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 25 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 12 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 12 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 33 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 40 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 34 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 53 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 57 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 318 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 336 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 182 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 500 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 107 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 184 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 483 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 314 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 338 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 5.4 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 120 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 79 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 10 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 5.8 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 4.6 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 11 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 160 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 1.4 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 136 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 354 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 564 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 186 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 1.7 MiB |
Binary file not shown.
|
After Width: | Height: | Size: 513 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 1.2 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 185 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 131 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 41 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 136 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 158 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 146 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 131 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 117 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 115 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 120 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 133 KiB |
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue