litmus-docs/website/versioned_docs/version-1.7.0/disk-fill.md

id	title	sidebar_label	original_id
disk-fill	Disk Fill Experiment Details	Disk Fill	disk-fill

---

Experiment Metadata

Type	Description	Tested K8s Platform
Chaos	Fill up Ephemeral Storage of a Pod	GKE, EKS, AKS

Prerequisites

• Ensure that Kubernetes Version > 1.13
• Ensure that the Litmus Chaos Operator is running by executing kubectl get pods in operator namespace (typically, litmus). If not, install from here
• Ensure that the disk-fill experiment resource is available in the cluster by executing kubectl get chaosexperiments in the desired namespace If not, install from here
• Cluster must run docker container runtime
• Appropriate Ephemeral Storage Requests and Limits should be set for the application before running the experiment. An example specification is shown below:

apiVersion: v1
kind: Pod
metadata:
  name: frontend
spec:
  containers:
  - name: db
    image: mysql
    env:
    - name: MYSQL_ROOT_PASSWORD
      value: "password"
    resources:
      requests:
        ephemeral-storage: "2Gi"
      limits:
        ephemeral-storage: "4Gi"
  - name: wp
    image: wordpress
    resources:
      requests:
        ephemeral-storage: "2Gi"
      limits:
        ephemeral-storage: "4Gi"

Entry-Criteria

• Application pods are healthy before chaos injection.

Exit-Criteria

• Application pods are healthy post chaos injection.

Details

• Causes Disk Stress by filling up the ephemeral storage of the pod (in the /var/lib/docker/container/{{container_id}}) on any given node.
• Causes the application pod to get evicted if the capacity filled exceeds the pod's ephemeral storage limit.
• Tests the Ephemeral Storage Limits, to ensure those parameters are sufficient.
• Tests the application's resiliency to disk stress/replica evictions.

Integrations

• Disk Fill can be effected using the chaos library: litmus, which makes use of dd to create a file of specified capacity on the node.
• The desired chaoslib can be selected by setting the above options as value for the env variable LIB

Steps to Execute the Chaos Experiment

• This Chaos Experiment can be triggered by creating a ChaosEngine resource on the cluster. To understand the values to provide in a ChaosEngine specification, refer Getting Started

• Follow the steps in the sections below to create the chaosServiceAccount, prepare the ChaosEngine & execute the experiment.

Prepare chaosServiceAccount

• Use this sample RBAC manifest to create a chaosServiceAccount in the desired (app) namespace. This example consists of the minimum necessary role permissions to execute the experiment.

Sample Rbac Manifest

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: disk-fill-sa
  namespace: default
  labels:
    name: disk-fill-sa
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: disk-fill-sa
  labels:
    name: disk-fill-sa
rules:
  - apiGroups: ["", "apps", "litmuschaos.io", "batch"]
    resources:
      [
        "pods",
        "jobs",
        "pods/exec",
        "events",
        "pods/log",
        "chaosengines",
        "chaosexperiments",
        "chaosresults",
      ]
    verbs: ["create", "list", "get", "patch", "update", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: disk-fill-sa
  labels:
    name: disk-fill-sa
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: disk-fill-sa
subjects:
  - kind: ServiceAccount
    name: disk-fill-sa
    namespace: default

Prepare ChaosEngine

• Provide the application info in spec.appinfo
• Provide the auxiliary applications info (ns & labels) in spec.auxiliaryAppInfo
• Override the experiment tunables if desired in experiments.spec.components.env
• To understand the values to provided in a ChaosEngine specification, refer ChaosEngine Concepts

Supported Experiment Tunables

Variables	Description	Specify In ChaosEngine	Notes
FILL_PERCENTAGE	Percentage to fill the Ephemeral storage limit	Mandatory	Can be set to more than 100 also, to force evict the pod
TARGET_CONTAINER	Name of container which is subjected to disk-fill	Mandatory
CONTAINER_PATH	Storage Location of containers	Optional	Defaults to '/var/lib/docker/containers'
TOTAL_CHAOS_DURATION	The time duration for chaos insertion (sec)	Optional	Defaults to 60s
LIB	The chaos lib used to inject the chaos	Optional	Defaults to `litmus`
LIB_IMAGE	The image used to fill the disk	Optional	Defaults to `litmuschaos/litmus-go:latest`
RAMP_TIME	Period to wait before injection of chaos in sec	Optional
INSTANCE_ID	A user-defined string that holds metadata/info about current run/instance of chaos. Ex: 04-05-2020-9-00. This string is appended as suffix in the chaosresult CR name.	Optional	Ensure that the overall length of the chaosresult CR is still < 64 characters

Sample ChaosEngine Manifest

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: nginx-chaos
  namespace: default
spec:
  # It can be true/false
  annotationCheck: "false"
  # It can be active/stop
  engineState: "active"
  #ex. values: ns1:name=percona,ns2:run=nginx
  auxiliaryAppInfo: ""
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: disk-fill-sa
  monitoring: false
  # It can be delete/retain
  jobCleanUpPolicy: "delete"
  experiments:
    - name: disk-fill
      spec:
        components:
          env:
            # specify the fill percentage according to the disk pressure required
            - name: FILL_PERCENTAGE
              value: "80"

            - name: TARGET_CONTAINER
              value: "nginx"

Create the ChaosEngine Resource

• Create the ChaosEngine manifest prepared in the previous step to trigger the Chaos.

  kubectl apply -f chaosengine.yml

• If the chaos experiment is not executed, refer to the troubleshooting section to identify the root cause and fix the issues.

Watch Chaos progress

• View the status of the pods as they are subjected to disk stress.

  watch -n 1 kubectl get pods -n <application-namespace>

• Monitor the capacity filled up on the host filesystem

  watch -n 1 du -kh /var/lib/docker/containers/<container-id>

Abort/Restart the Chaos Experiment

• To stop the pod-delete experiment immediately, either delete the ChaosEngine resource or execute the following command:

  kubectl patch chaosengine <chaosengine-name> -n <namespace> --type merge --patch '{"spec":{"engineState":"stop"}}'

• To restart the experiment, either re-apply the ChaosEngine YAML or execute the following command:

  kubectl patch chaosengine <chaosengine-name> -n <namespace> --type merge --patch '{"spec":{"engineState":"active"}}'

Notes:

• The abort will stop further fill of the local disk, but doesn't reclaim the used capacity. This is a manual operation as of today. The auto-rollback, i.e., in this case reclaim of currently filled disk-space will be implemented in a future release

• However, upon graceful completion of the experiment (i.e.,un-aborted), the space is automatically reclaimed as the chaos impact is reverted.

Check Chaos Experiment Result

• Check whether the application is resilient to the container kill, once the experiment (job) is completed. The ChaosResult resource name is derived like this: <ChaosEngine-Name>-<ChaosExperiment-Name>.

  kubectl describe chaosresult nginx-chaos-disk-fill -n <application-namespace>

Disk Fill Experiment Demo

• A sample recording of this experiment execution is provided here.