litmuschaos
/
litmus-docs
mirror of https://github.com/litmuschaos/litmus-docs.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
litmus-docs/website/versioned_docs/version-1.10.0/container-kill.md

10 KiB
Raw Blame History

id title sidebar_label original_id
container-kill Container Kill Experiment Details Container Kill container-kill

Experiment Metadata

Type Description Tested K8s Platform
Generic Kill one container in the application pod GKE, Packet(Kubeadm), Minikube, EKS, AKS

Prerequisites

  • Ensure that the Litmus ChaosOperator is running by executing kubectl get pods in operator namespace (typically, litmus). If not, install from here
  • Ensure that the container-kill experiment resource is available in the cluster by executing kubectl get chaosexperiments in the desired namespace. If not, install from here

Entry Criteria

  • Application pods are healthy before chaos injection

Exit Criteria

  • Application pods are healthy post chaos injection

Details

  • litmus lib in docker runtime details
    • It can kill the container of multiple pods in parallel (can be tuned by PODS_AFFECTED_PERC env). It kill the container by sending SIGKILL termination signal to its docker socket (hence docker runtime is required)
    • Containers are killed using the kill command provided by pumba
    • Pumba is run as a pod on the application node. It have ability to kill the application containers multiple times. Which can be varied by TOTAL_CHAOS_DURATION and CHAOS_INTERVAL.
  • litmus chaoslib in containerd and crio runtime codetails
    • It can kill the container of multiple pods in parallel (can be tuned by PODS_AFFECTED_PERC env).
    • Containers are killed using the crictl stop command.
    • container-kill is run as a pod on the application node. It have ability to kill the application containers multiple times. Which can be varied by TOTAL_CHAOS_DURATION and CHAOS_INTERVAL.
  • Tests deployment sanity (replica availability & uninterrupted service) and recovery workflow of the application
  • Good for testing recovery of pods having side-car containers

Integrations

  • Container kill is achieved using the litmus chaos library
  • The container runtime can be choose via setting CONTAINER_RUNTIME env. supported values: docker, containerd, crio
  • The desired pumba and litmus image can be configured in the env variable LIB_IMAGE.

Steps to Execute the ChaosExperiment

  • This ChaosExperiment can be triggered by creating a ChaosEngine resource on the cluster. To understand the values to provide in a ChaosEngine specification, refer Getting Started

  • Follow the steps in the sections below to create the chaosServiceAccount, prepare the ChaosEngine & execute the experiment.

Prepare chaosServiceAccount

  • Use this sample RBAC manifest to create a chaosServiceAccount in the desired (app) namespace. This example consists of the minimum necessary role permissions to execute the experiment.

Sample Rbac Manifest

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: container-kill-sa
  namespace: default
  labels:
    name: container-kill-sa
    app.kubernetes.io/part-of: litmus
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: container-kill-sa
  namespace: default
  labels:
    name: container-kill-sa
    app.kubernetes.io/part-of: litmus
rules:
  - apiGroups: ["", "litmuschaos.io", "batch", "apps"]
    resources:
      [
        "pods",
        "jobs",
        "pods/exec",
        "pods/log",
        "events",
        "chaosengines",
        "chaosexperiments",
        "chaosresults",
      ]
    verbs:
      ["create", "list", "get", "patch", "update", "delete", "deletecollection"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: container-kill-sa
  namespace: default
  labels:
    name: container-kill-sa
    app.kubernetes.io/part-of: litmus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: container-kill-sa
subjects:
  - kind: ServiceAccount
    name: container-kill-sa
    namespace: default

Prepare ChaosEngine

  • Provide the application info in spec.appinfo
  • Override the experiment tunables if desired in experiments.spec.components.env
  • To understand the values to provided in a ChaosEngine specification, refer ChaosEngine Concepts

Note: Ensure that the CHAOS_INTERVAL used in the experiment is fairly high (i.e., enough time is provided for app recovery). Else, it can cause an exponential increase in the backOff delay, causing the app to stay in CrashLoopBackOff state for much longer

Supported Experiment Tunables

Variables Description Specify In ChaosEngine Notes
TARGET_CONTAINER The container to be killed inside the pod Mandatory If the TARGET_CONTAINER is not provided it will delete the first container
CHAOS_INTERVAL Time interval b/w two successive container kill (in sec) Optional If the CHAOS_INTERVAL is not provided it will take the default value of 10s
TOTAL_CHAOS_DURATION The time duration for chaos injection (seconds) Optional Defaults to 20s
PODS_AFFECTED_PERC The Percentage of total pods to target Optional Defaults to 0% (corresponds to 1 replica)
TARGET_PODS Comma separated list of application pod name subjected to container kill chaos Optional If not provided, it will select target pods randomly based on provided appLabels
LIB_IMAGE LIB Image used to kill the container Optional Defaults to `litmuschaos/go-runner:latest`
LIB The category of lib use to inject chaos Optional Default value: litmus, only litmus supported
RAMP_TIME Period to wait before injection of chaos in sec Optional
SEQUENCE It defines sequence of chaos execution for multiple target pods Optional Default value: parallel. Supported: serial, parallel
SOCKET_PATH Path of the containerd/crio socket file Optional Defaults to `/run/containerd/containerd.sock`
CONTAINER_RUNTIME container runtime interface for the cluster Optional Defaults to docker, supported values: docker, containerd, crio
INSTANCE_ID A user-defined string that holds metadata/info about current run/instance of chaos. Ex: 04-05-2020-9-00. This string is appended as suffix in the chaosresult CR name. Optional Ensure that the overall length of the chaosresult CR is still < 64 characters

Sample ChaosEngine Manifest

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: nginx-chaos
  namespace: default
spec:
  # It can be true/false
  annotationCheck: "true"
  # It can be active/stop
  engineState: "active"
  #ex. values: ns1:name=percona,ns2:run=nginx
  auxiliaryAppInfo: ""
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: container-kill-sa
  monitoring: false
  # It can be delete/retain
  jobCleanUpPolicy: "delete"
  experiments:
    - name: container-kill
      spec:
        components:
          env:
            # specify the name of the container to be killed
            - name: TARGET_CONTAINER
              value: "nginx"

            # provide the chaos interval
            - name: CHAOS_INTERVAL
              value: "10"

            # provide the total chaos duration
            - name: TOTAL_CHAOS_DURATION
              value: "20"

            # provide the name of container runtime
            # it supports docker, containerd, crio
            # default to docker
            - name: CONTAINER_RUNTIME
              value: "docker"

            # provide the socket file path
            # applicable only for containerd runtime
            - name: SOCKET_PATH
              value: "/run/containerd/containerd.sock"

Create the ChaosEngine Resource

  • Create the ChaosEngine manifest prepared in the previous step to trigger the Chaos.

    kubectl apply -f chaosengine.yml

  • If the chaos experiment is not executed, refer to the troubleshooting section to identify the root cause and fix the issues.

Watch Chaos progress

  • View pod restart count by setting up a watch on the pods in the application namespace

    watch -n 1 kubectl get pods -n <application-namespace>

Abort/Restart the ChaosExperiment

  • To stop the pod-delete experiment immediately, either delete the ChaosEngine resource or execute the following command:

    kubectl patch chaosengine <chaosengine-name> -n <namespace> --type merge --patch '{"spec":{"engineState":"stop"}}'

  • To restart the experiment, either re-apply the ChaosEngine YAML or execute the following command:

    kubectl patch chaosengine <chaosengine-name> -n <namespace> --type merge --patch '{"spec":{"engineState":"active"}}'

Check ChaosExperiment Result

  • Check whether the application is resilient to the container kill, once the experiment (job) is completed. The ChaosResult resource name is derived like this: <ChaosEngine-Name>-<ChaosExperiment-Name>.

    kubectl describe chaosresult nginx-chaos-container-kill -n <application-namespace>

Application Container Kill Demo

  • A sample recording of this experiment execution is provided here.