5.7 KiB

Raw Blame History

id	title	sidebar_label	original_id
container-kill	Container Kill Experiment Details	Container Kill	container-kill

## Experiment Metadata | Type | Description | Tested K8s Platform | | ----------| ------------------------ | ------------------------------------------------------------------| | Generic | Kill one container in the application pod | GKE, Packet(Kubeadm), Minikube| ## Prerequisites - Ensure that the Litmus Chaos Operator is running by executing kubectl get pods in operator namespace (typically, litmus). If not, install from here - Ensure that the container-kill experiment resource is available in the cluster by executing kubectl get chaosexperiments in the desired namespace. If not, install from here - Cluster must run docker container runtime ## Entry Criteria - Application pods are healthy before chaos injection ## Exit Criteria - Application pods are healthy post chaos injection ## Details - Kills one container in the specified application pod by sending SIGKILL termination signal to its docker socket (hence docker runtime is required) - Containers are killed using the kill command provided by pumba - Pumba is run as a daemonset on all nodes in dry-run mode to begin with; the kill command is issued during experiment execution via kubectl exec - Tests deployment sanity (replica availability & uninterrupted service) and recovery workflow of the application - Good for testing recovery of pods having side-car containers ## Integrations - Container kill is achieved using the pumba chaos library - The desired pumba image can be configured in the env variable LIB_IMAGE. ## Steps to Execute the Chaos Experiment - This Chaos Experiment can be triggered by creating a ChaosEngine resource on the cluster. To understand the values to provide in a ChaosEngine specification, refer Getting Started - Follow the steps in the sections below to create the chaosServiceAccount, prepare the ChaosEngine & execute the experiment. ### Prepare chaosServiceAccount - Use this sample RBAC manifest to create a chaosServiceAccount in the desired (app) namespace. This example consists of the minimum necessary role permissions to execute the experiment. #### Sample Rbac Manifest ```yaml apiVersion: v1 kind: ServiceAccount metadata: name: nginx-sa namespace: default labels: name: nginx-sa

Source: openebs/templates/clusterrole.yaml

apiVersion: rbac.authorization.k8s.io/v1beta1 kind: Role metadata: name: nginx-sa labels: name: nginx-sa rules:

apiGroups: ["","litmuschaos.io","batch","apps"] resources: ["pods","jobs","daemonsets","pods/exec","chaosengines","chaosexperiments","chaosresults"] verbs: ["create","list","get","patch","update","delete"]

apiVersion: rbac.authorization.k8s.io/v1beta1 kind: RoleBinding metadata: name: nginx-sa labels: name: nginx-sa roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: nginx-sa subjects:

kind: ServiceAccount name: nginx-sa namespace: default


### Prepare ChaosEngine

- Provide the application info in `spec.appinfo`
- Override the experiment tunables if desired

#### Supported Experiment Tunables

<table>
<tr>
<th> Variables </th>
<th> Description  </th>
<th> Type </th>
<th> Notes </th>
</tr>
<tr>
<td> TARGET_CONTAINER  </td>
<td> The container to be killed inside the pod </td>
<td> Mandatory </td>
<td> If the TARGET_CONTAINER is not provided it will delete the first container </td>
</tr>
<tr>
<td> LIB_IMAGE  </td>
<td> The pumba image used to run the kill command </td>
<td> Optional </td>
<td> Defaults to `gaiaadm/pumba:0.4.8`. Note: pumba images >=0.6 do not work with this experiment. </td>
</tr>
<tr>
<td> LIB  </td>
<td> The category of lib use to inject chaos </td>
<td> Optional  </td>
<td> Only `pumba` supported currently </td>
</tr>
</table>


#### Sample ChaosEngine Manifest

```yaml
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: nginx-chaos
  namespace: default
spec:
  # It can be app/infra
  chaosType: 'app'
  #ex. values: ns1:name=percona,ns2:run=nginx 
  auxiliaryAppInfo: ""
  appinfo:
    appns: default
    applabel: 'app=nginx'
    appkind: deployment
  chaosServiceAccount: nginx-sa
  monitoring: false
  components:
    runner:
      image: "litmuschaos/chaos-executor:1.0.0"
      type: "go"
  # It can be delete/retain
  jobCleanUpPolicy: delete 
  experiments:
    - name: container-kill
      spec:
        components:
           # specify the name of the container to be killed
          - name: TARGET_CONTAINER
            value: 'nginx'

Create the ChaosEngine Resource

Create the ChaosEngine manifest prepared in the previous step to trigger the Chaos.

kubectl apply -f chaosengine.yml

Watch Chaos progress

View pod restart count by setting up a watch on the pods in the application namespace

watch -n 1 kubectl get pods -n <application-namespace>

Check Chaos Experiment Result

Check whether the application is resilient to the container kill, once the experiment (job) is completed. The ChaosResult resource name is derived like this: <ChaosEngine-Name>-<ChaosExperiment-Name>.

kubectl describe chaosresult nginx-chaos-container-kill -n <application-namespace>

Application Container Kill Demo

A sample recording of this experiment execution is provided here.

5.7 KiB Raw Blame History