(feat)developer_docs: add tool to scaffold experiment artefacts (#887)
Signed-off-by: ksatchit <ksatchit@mayadata.io>
This commit is contained in:
Karthik Satchitanand
GitHub
parent
0f43128bc1
commit
84cd26daed
|
|
@ -0,0 +1,144 @@
|
|||
## Steps to Bootstrap a Chaos Experiment
|
||||
|
||||
(Ref: [Contributing a Chaos Chart](https://github.com/litmuschaos/chaos-charts/blob/master/CONTRIBUTING.md))
|
||||
|
||||
The artefacts associated with a chaos-experiment are summarized below:
|
||||
|
||||
- Submitted in the litmuschaos/litmus repository, under the experiments/*chaos category* folder
|
||||
|
||||
- Experiment business logic (either as ansible playbook, or other). May involve creation of new or reuse of existing chaoslib
|
||||
- Experiment Kubernetes job (executes the business logic, also called litmusbook)
|
||||
|
||||
Example: [pod delete experiment in litmus](/experiments/generic/pod_delete)
|
||||
|
||||
- Submitted in litmuschaos/chaos-charts repository, under the *chaos category* folder
|
||||
|
||||
- Experiment custom resource (CR) (holds experiment-specific chaos parameters & playbook entrypoint)
|
||||
- Experiment ChartServiceVersion (holds experiment metadata that will be rendered on [charthub](hub.litmuschaos.io))
|
||||
|
||||
Example: [pod delete experiment in chaos-charts](https://github.com/litmuschaos/chaos-charts/tree/master/charts/generic/pod-delete)
|
||||
|
||||
The *generate_charts.py* script is a simple way to bootstrap your experiment, and helps create the aforementioned artefacts in the
|
||||
appropriate directory (i.e., as per the chaos-category) based on an attributes file provided as input by the chart-developer. The
|
||||
scaffolded files consist of placeholders which can then be filled as desired.
|
||||
|
||||
### Pre-Requisites
|
||||
|
||||
- *python3* is available (`sudo apt-get install python3.6`)
|
||||
- *jinja2* & *pyYaml* python packages are available (`sudo apt-get install python3-pip`, `pip install jinja2`, `pip install pyYaml`)
|
||||
|
||||
### Steps to Generate Experiment Manifests
|
||||
|
||||
- Clone the litmus repository & navigate to the `contribute/developer_guide` folder
|
||||
|
||||
```
|
||||
$ git clone https://github.com/litmuschaos/litmus.git
|
||||
$ cd litmus/contribute/developer_guide
|
||||
```
|
||||
|
||||
- Populate the `attributes.yaml` with details of the chaos experiment (or chart). Use the [attributes.yaml.sample](/contribute/developer_guide/attributes.yaml.sample) as reference.
|
||||
|
||||
As an example, let us consider an experiment to kill one of the replicas of a nodejs-app that provides a simple
|
||||
hello-world service. The attributes.yaml can be constructed like this:
|
||||
|
||||
```
|
||||
$ cat attributes.yaml
|
||||
|
||||
---
|
||||
name: kill-hello-replicas
|
||||
version: 0.1.0
|
||||
category: hello-world
|
||||
repository: https://github.com/litmuschaos/demo-app/tree/master/sample_apps/hellonode
|
||||
community: https://kubernetes.slack.com/messages/CNXNB0ZTN
|
||||
description: "kills hello-world pods in a random manner"
|
||||
keywords:
|
||||
- pods
|
||||
- kubernetes
|
||||
- hello-world
|
||||
- nodejs
|
||||
maturity: alpha
|
||||
maintainers:
|
||||
- ksatchit@mayadata.io
|
||||
contributors:
|
||||
- ksatchit@mayadata.io
|
||||
min_kubernetes_version: 1.12.0
|
||||
references:
|
||||
- https://docs.litmuschaos.io/docs/getstarted/
|
||||
```
|
||||
|
||||
- Run the following command to generate the necessary artefacts for submitting the `hello-world` chaos chart with
|
||||
`kill-hello-replicas` experiment.
|
||||
|
||||
```
|
||||
$ python3 generate_chart.py --attributes_file=attributes.yaml --generate_type=experiment
|
||||
```
|
||||
View the generated files
|
||||
|
||||
```
|
||||
$ ls -ltr
|
||||
|
||||
total 12
|
||||
drwxr-xr-x 2 ksatchit ksatchit 4096 Oct 17 12:54 mysql
|
||||
drwxr-xr-x 6 ksatchit ksatchit 4096 Oct 21 09:28 generic
|
||||
drwxr-xr-x 3 ksatchit ksatchit 4096 Oct 21 10:54 hello-world
|
||||
|
||||
$ ls -ltr hello-world/
|
||||
|
||||
total 8
|
||||
-rw-r--r-- 1 ksatchit ksatchit 704 Oct 21 10:54 hello-world.chartserviceversion.yaml
|
||||
drwxr-xr-x 2 ksatchit ksatchit 4096 Oct 21 10:54 kill-hello-replicas
|
||||
|
||||
$ ls -ltr hello-world/kill-hello-replicas
|
||||
|
||||
total 20
|
||||
-rw-r--r-- 1 ksatchit ksatchit 704 Oct 21 10:54 kill-hello-replicas.chartserviceversion.yaml
|
||||
-rw-r--r-- 1 ksatchit ksatchit 1405 Oct 21 10:54 kill-hello-replicas-k8s-job.yml
|
||||
-rw-r--r-- 1 ksatchit ksatchit 590 Oct 21 10:54 kill-hello-replicas-experiment-cr.yml
|
||||
-rw-r--r-- 1 ksatchit ksatchit 69 Oct 21 10:54 kill-hello-replicas-ansible-prerequisites.yml
|
||||
-rw-r--r-- 1 ksatchit ksatchit 2167 Oct 21 10:54 kill-hello-replicas-ansible-logic.yml
|
||||
|
||||
```
|
||||
|
||||
**Note**: In the `--generate_type` attribute, select the appropriate type of manifests to be generated, where,
|
||||
- `chart`: Just the chaos-chart metadata, i.e., chartserviceversion yaml
|
||||
- `experiment`: Chaos experiment artefacts belonging to a an existing OR new chart.
|
||||
|
||||
- Proceed with construction of business logic inside the `kill-hello-replicas-ansible-logic.yml` file, by making
|
||||
the appropriate modifications listed below to achieve the desired effect:
|
||||
|
||||
- variables
|
||||
- entry & exit criteria checks for the experiment
|
||||
- helper utils in either [utils](/utils/) or new [base chaos libraries](/chaoslib)
|
||||
|
||||
- Update the `kill-hello-replicas-experiment-cr.yml` with the right chaos params in the `spec.definition.env` with their
|
||||
default values
|
||||
|
||||
- Create an experiment README (example: [pod delete readme](experiments/generic/pod_delete/README.md)) explaining, briefly,
|
||||
the *what*, *why* & *how* of the experiment to aid users of this experiment.
|
||||
|
||||
### Steps to Test Experiment
|
||||
|
||||
The experiment created using the above steps, can be tested in the following manner:
|
||||
|
||||
- Run the `kill-hello-replicas-k8s-job.yml` with the desired values in the ENV and appropriate `chaosServiceAccount`
|
||||
using a custom dev image instead of `litmuschaos/ansible-runner` (say, ksatchit/ansible-runner) that packages the
|
||||
business logic.
|
||||
|
||||
- (Optional) Once the experiment has been validated using the above step, it can also be tested against the standard chaos
|
||||
workflow using the `kill-hello-replicas-experiment-cr.yml`. This involves:
|
||||
|
||||
- Launching the Chaos-Operator
|
||||
- Creating the ChaosExperiment CR on the cluster (use the same custom dev image used in the above step)
|
||||
- Creating the ChaosEngine to execute the above ChaosExperiment
|
||||
- Verifying the experiment status via ChaosResult
|
||||
|
||||
Refer [litmus docs](https://docs.litmuschaos.io/docs/getstarted/) for more details on this procedure.
|
||||
|
||||
### Steps to Include the Chaos Charts/Experiments into the ChartHub
|
||||
|
||||
- Send a PR to the [litmus](https://github.com/litmuschaos/litmus) repo with the modified litmusbook files
|
||||
- Send a PR to the [chaos-charts](https://github.com/litmuschaos/chaos-charts) repo with the modified experiment CR,
|
||||
experiment chartserviceversion, chaos chart (category-level) chartserviceversion & package (if applicable) YAMLs
|
||||
|
||||
|
||||
|
||||
|
|
@ -0,0 +1,21 @@
|
|||
---
|
||||
name: kill-hello-replicas
|
||||
version: 0.1.0
|
||||
category: hello-world
|
||||
repository: https://github.com/litmuschaos/demo-app/tree/master/sample_apps/hellonode
|
||||
community: https://kubernetes.slack.com/messages/CNXNB0ZTN
|
||||
description: "kills hello-world pods in a random manner"
|
||||
keywords:
|
||||
- pods
|
||||
- kubernetes
|
||||
- hello-world
|
||||
- nodejs
|
||||
maturity: alpha
|
||||
maintainers:
|
||||
- ksatchit@mayadata.io
|
||||
contributors:
|
||||
- ksatchit@mayadata.io
|
||||
min_kubernetes_version: 1.12.0
|
||||
references:
|
||||
- https://docs.litmuschaos.io/docs/getstarted/
|
||||
|
||||
|
|
@ -0,0 +1,133 @@
|
|||
from jinja2 import Environment, FileSystemLoader
|
||||
import yaml
|
||||
import os
|
||||
import sys
|
||||
import argparse
|
||||
|
||||
'''
|
||||
TODO:
|
||||
1. No try-catch blocks have been used to account for empty/uninitialized params in attributes.yaml
|
||||
|
||||
2. Add basic unit/bdd tests (manually checked currently) & necessary code refactor to facilitate this:
|
||||
|
||||
a. (positive) Verify creation of CSV for --generate_type=chart
|
||||
b. (positive) Verify creation of CSV using category name for --generate_type=experiment
|
||||
c. (positive) Verify creation of business logic, job & experiment CRs for --generate_type=experiment
|
||||
d. (negative) Verify validation/err-handle upon empty name, category attributes
|
||||
'''
|
||||
|
||||
'''
|
||||
NOTES:
|
||||
1. Category attribute is expected to match with chart names(though not mandatory), as per convention
|
||||
followed in litmuschaos/chaos-charts
|
||||
'''
|
||||
|
||||
|
||||
""" generate_csv creates the chartserviceversion manifest """
|
||||
def generate_csv(csv_parent_path, csv_name, csv_config, litmus_env):
|
||||
csv_filename = csv_parent_path + '/' + csv_name + '.' + 'chartserviceversion.yaml'
|
||||
|
||||
# Load Jinja2 template
|
||||
template = litmus_env.get_template('./templates/chartserviceversion.tmpl')
|
||||
output_from_parsed_template = template.render(csv_config)
|
||||
with open(csv_filename, "w+") as f:
|
||||
f.write(output_from_parsed_template)
|
||||
|
||||
""" generate_package creates the package manifest """
|
||||
def generate_package(package_parent_path, package_name):
|
||||
package_filename = package_parent_path + '/' + package_name + '.' + 'package.yaml'
|
||||
print(package_filename)
|
||||
with open(package_filename, "w+") as f:
|
||||
f.write('packageName: ' + package_name + '\n' + 'experiments:')
|
||||
|
||||
""" generate_litmusbook creates the experiment business logic, kubernetes job & custom resource manifests """
|
||||
def generate_litmusbook(litmusbook_parent_path, litmusbook_name, litmus_env):
|
||||
|
||||
k8s_job_filename = litmusbook_parent_path + '/' + litmusbook_name + '-' + 'k8s-job.yml'
|
||||
chaos_logic_filename = litmusbook_parent_path + '/' + litmusbook_name + '-' + 'ansible-logic.yml'
|
||||
chaos_prerequisites_filename = litmusbook_parent_path + '/' + litmusbook_name + '-' + 'ansible-prerequisites.yml'
|
||||
chaos_experiment_cr_filename = litmusbook_parent_path + '/' + litmusbook_name + '-' + 'experiment-cr.yml'
|
||||
|
||||
k8s_job_template = litmus_env.get_template('./templates/experiment_k8s_job.tmpl')
|
||||
chaos_logic_template = litmus_env.get_template('./templates/experiment_ansible_logic.tmpl')
|
||||
chaos_prerequisites_template = litmus_env.get_template('./templates/experiment_ansible_prerequisites.tmpl')
|
||||
chaos_experiment_cr_template = litmus_env.get_template('./templates/experiment_custom_resource.tmpl')
|
||||
|
||||
# create a dictionary with artifact as key & list of rendered jinja template & filepath as value
|
||||
litmusbook_dict = {
|
||||
"job": [k8s_job_template, k8s_job_filename],
|
||||
"chaos_logic": [chaos_logic_template, chaos_logic_filename],
|
||||
"chaos_prerequisites": [chaos_prerequisites_template, chaos_prerequisites_filename],
|
||||
"chaos_experiment_cr": [chaos_experiment_cr_template, chaos_experiment_cr_filename]
|
||||
}
|
||||
|
||||
# derive the chart directory in order to use in the generated k8s job spec
|
||||
chart_dir = litmusbook_parent_path.split("/")[-2]
|
||||
print(litmusbook_parent_path,chart_dir)
|
||||
|
||||
for artifact in litmusbook_dict.values():
|
||||
output_from_parsed_template = artifact[0].render(name=litmusbook_name, chart=chart_dir)
|
||||
with open(artifact[1], "w+") as f:
|
||||
f.write(output_from_parsed_template)
|
||||
|
||||
def main():
|
||||
# Required Arguments
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("-a", "--attributes_file", required=True,
|
||||
help="metadata to generate chartserviceversion yaml")
|
||||
parser.add_argument("-t", "--generate_type", required=True,
|
||||
help="scaffold a new chart or experiment into existing chart")
|
||||
# Optional Arguments
|
||||
parser.add_argument("-c", "--chart_name", required=False,
|
||||
help="existing chart name to which experiment belongs, defaults to 'category' in attributes file")
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
entity_metadata_source = args.attributes_file
|
||||
entity_type = args.generate_type
|
||||
entity_parent = args.chart_name
|
||||
|
||||
# Load data from YAML file into a dictionary
|
||||
config = yaml.load(open(entity_metadata_source))
|
||||
entity_name = config['name']
|
||||
|
||||
# Store the litmus root from bootstrap folder
|
||||
litmus_root = path = os.path.abspath(os.path.join("..", os.pardir))
|
||||
env = Environment(loader = FileSystemLoader('./'), trim_blocks=True, lstrip_blocks=True)
|
||||
|
||||
# if generate_type is chart, only create the chart(top)-level CSV & package manifests
|
||||
if entity_type == 'chart':
|
||||
chart_dir = litmus_root + '/experiments/' + entity_name
|
||||
if os.path.isdir(chart_dir) != True:
|
||||
os.makedirs(chart_dir)
|
||||
generate_csv(chart_dir, entity_name, config, env)
|
||||
generate_package(chart_dir, entity_name)
|
||||
|
||||
# if generate_type is experiment, create the litmusbook arefacts (job, playbook, cr)
|
||||
elif entity_type == 'experiment':
|
||||
# if chart_name is not explicitly provided, use "category" from attributes.yaml as chart
|
||||
if entity_parent is None:
|
||||
experiment_category = config['category']
|
||||
chart_dir = litmus_root + '/experiments/' + experiment_category
|
||||
else:
|
||||
chart_dir = litmus_root + '/experiments/' + entity_parent
|
||||
# if a folder with specified/derived chart name is not present, create it
|
||||
if os.path.isdir(chart_dir) != True:
|
||||
os.makedirs(chart_dir)
|
||||
# generate chart csv & package for the freshly created chart folder
|
||||
generate_csv(chart_dir, experiment_category, config, env)
|
||||
generate_package(chart_dir, experiment_category)
|
||||
|
||||
# create experiment folder inside the chart folder
|
||||
experiment_dir = chart_dir + '/' + entity_name
|
||||
if os.path.isdir(experiment_dir) != True:
|
||||
os.makedirs(experiment_dir)
|
||||
|
||||
# generate experiment csv
|
||||
generate_csv(experiment_dir, entity_name, config, env)
|
||||
# generate job, playbook and cr spec from templates
|
||||
generate_litmusbook(experiment_dir, entity_name, env)
|
||||
|
||||
|
||||
if __name__=="__main__":
|
||||
main()
|
||||
|
|
@ -0,0 +1,22 @@
|
|||
apiVersion: litmuchaos.io/v1alpha1
|
||||
kind: ChartServiceVersion
|
||||
metadata:
|
||||
name: {{ name }}
|
||||
version: {{ version }}
|
||||
annotations:
|
||||
categories: {{ category }}
|
||||
repository: {{ repository }}
|
||||
support: {{ community }}
|
||||
spec:
|
||||
displayName: {{ name }}
|
||||
description: >
|
||||
{{ description }}
|
||||
keywords: {{ keywords }}
|
||||
maturity: {{ maturity }}
|
||||
minKubeVersion: {{ min_kubernetes_version }}
|
||||
maintainers: {{ maintainers }}
|
||||
contributors: {{ contributors }}
|
||||
links: {{ references }}
|
||||
icon:
|
||||
- url:
|
||||
mediatype: ""
|
||||
|
|
@ -0,0 +1,75 @@
|
|||
---
|
||||
- hosts: localhost
|
||||
connection: local
|
||||
|
||||
vars:
|
||||
c_experiment: {{ name }}
|
||||
a_ns: "{{ '{{' }} lookup('env','APP_NAMESPACE') {{ '}}' }}"
|
||||
a_label: "{{ '{{' }} lookup('env','APP_LABEL') {{ '}}' }}"
|
||||
a_kind: "{{ '{{' }} lookup('env','APP_KIND') {{ '}}' }}"
|
||||
c_duration: "{{ '{{' }} lookup('env','TOTAL_CHAOS_DURATION') {{ '}}' }}"
|
||||
c_util: ""
|
||||
|
||||
tasks:
|
||||
- block:
|
||||
|
||||
- include: {{ name }}-ansible-prerequisites.yml
|
||||
|
||||
## GENERATE EXP RESULT NAME
|
||||
- block:
|
||||
|
||||
- name: Construct chaos result name (experiment_name)
|
||||
set_fact:
|
||||
c_experiment: "{{ '{{' }} lookup('env','CHAOSENGINE') {{ '}}' }}-{{ '{{' }} c_experiment {{ '}}' }}"
|
||||
|
||||
when: lookup('env','CHAOSENGINE')
|
||||
|
||||
## RECORD START-OF-EXPERIMENT IN LITMUSCHAOS RESULT CR
|
||||
- include_tasks: /utils/runtime/update_chaos_result_resource.yml
|
||||
vars:
|
||||
status: 'SOT'
|
||||
namespace: "{{ '{{' }} a_ns {{ '}}' }}"
|
||||
|
||||
## PRE-CHAOS APPLICATION LIVENESS CHECK
|
||||
|
||||
- name: Verify that the AUT (Application Under Test) is running
|
||||
include_tasks: "/utils/common/status_app_pod.yml"
|
||||
vars:
|
||||
app_ns: "{{ '{{' }} a_ns {{ '}}' }}"
|
||||
app_lkey: "{{ '{{' }} a_label.split('=')[0] {{ '}}' }}"
|
||||
app_lvalue: "{{ '{{' }} a_label.split('=')[1] {{ '}}' }}"
|
||||
delay: 1
|
||||
retries: 60
|
||||
|
||||
## FAULT INJECTION
|
||||
|
||||
- include_tasks: "{{ '{{' }} c_util {{ '}}' }}"
|
||||
vars:
|
||||
c_svc_acc: "{{ '{{' }} lookup('env','CHAOS_SERVICE_ACCOUNT') {{ '}}' }}"
|
||||
|
||||
## POST-CHAOS APPLICATION LIVENESS CHECK
|
||||
|
||||
- name: Verify AUT liveness post fault-injection
|
||||
include_tasks: "/utils/common/status_app_pod.yml"
|
||||
vars:
|
||||
app_ns: "{{ '{{' }} a_ns {{ '}}' }}"
|
||||
app_lkey: "{{ '{{' }} a_label.split('=')[0] {{ '}}' }}"
|
||||
app_lvalue: "{{ '{{' }} a_label.split('=')[1] {{ '}}' }}"
|
||||
delay: 1
|
||||
retries: 60
|
||||
|
||||
- set_fact:
|
||||
flag: "pass"
|
||||
|
||||
rescue:
|
||||
- set_fact:
|
||||
flag: "fail"
|
||||
|
||||
always:
|
||||
|
||||
## RECORD END-OF-TEST IN LITMUSCHAOS RESULT CR
|
||||
- include_tasks: /utils/runtime/update_chaos_result_resource.yml
|
||||
vars:
|
||||
status: 'EOT'
|
||||
namespace: "{{ '{{' }} a_ns {{ '}}' }}"
|
||||
|
||||
|
|
@ -0,0 +1,5 @@
|
|||
---
|
||||
- name: placeholder-task-for-prerequisites
|
||||
debug:
|
||||
msg: ""
|
||||
|
||||
|
|
@ -0,0 +1,25 @@
|
|||
apiVersion: litmuschaos.io/v1alpha1
|
||||
description:
|
||||
message: |
|
||||
{{ description }}
|
||||
kind: ChaosExperiment
|
||||
metadata:
|
||||
name: {{ name }}
|
||||
version: {{ version }}
|
||||
spec:
|
||||
definition:
|
||||
image: "litmuschaos/ansible-runner:ci"
|
||||
args:
|
||||
- -c
|
||||
- ansible-playbook ./experiments/{{ chart }}/{{ name }}/{{ name }}-ansible-logic.yml -i /etc/ansible/hosts -vv; exit 0
|
||||
command:
|
||||
- /bin/bash
|
||||
env:
|
||||
- name: ANSIBLE_STDOUT_CALLBACK
|
||||
value: default
|
||||
- name: TOTAL_CHAOS_DURATION
|
||||
value: ""
|
||||
- name: LIB
|
||||
value: ""
|
||||
labels:
|
||||
experiment: {{ name }}
|
||||
|
|
@ -0,0 +1,51 @@
|
|||
---
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
generateName: {{ name }}-
|
||||
spec:
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
experiment: {{ name }}
|
||||
spec:
|
||||
# Placeholder that is updated by the executor for automated runs
|
||||
# Provide appropriate SA (with desired permissions) if executed manually
|
||||
serviceAccountName: %CHAOS_SERVICE_ACCOUNT%
|
||||
restartPolicy: Never
|
||||
containers:
|
||||
- name: ansibletest
|
||||
image: litmuschaos/ansible-runner:ci
|
||||
imagePullPolicy: Always
|
||||
env:
|
||||
- name: ANSIBLE_STDOUT_CALLBACK
|
||||
value: default
|
||||
|
||||
- name: APP_NAMESPACE
|
||||
value: ""
|
||||
|
||||
- name: APP_LABEL
|
||||
value: ""
|
||||
|
||||
- name: APP_KIND
|
||||
value: ""
|
||||
|
||||
- name: TOTAL_CHAOS_DURATION
|
||||
value: ""
|
||||
|
||||
## env var that describes the library used to execute the chaos
|
||||
## default: litmus. Supported values: litmus, powerfulseal, chaoskube
|
||||
- name: LIB
|
||||
value: ""
|
||||
|
||||
- name: CHAOSENGINE
|
||||
value: ""
|
||||
|
||||
- name: CHAOS_SERVICE_ACCOUNT
|
||||
valueFrom:
|
||||
fieldRef:
|
||||
fieldPath: spec.serviceAccountName
|
||||
|
||||
command: ["/bin/bash"]
|
||||
args: ["-c", "ansible-playbook ./experiments/{{ chart }}/{{ name }}/{{ name }}-ansible-logic.yml -i /etc/ansible/hosts; exit 0"]
|
||||
|
||||
Loading…
Reference in New Issue