--- id: disk-fill title: Disk Fill Experiment Details sidebar_label: Disk Fill original_id: disk-fill --- --- ## Experiment Metadata
Type Description Tested K8s Platform
Chaos Fill up Ephemeral Storage of a Pod GKE, EKS, AKS
## Prerequisites - Ensure that Kubernetes Version > 1.13 - Ensure that the Litmus Chaos Operator is running by executing `kubectl get pods` in operator namespace (typically, `litmus`). If not, install from [here](https://docs.litmuschaos.io/docs/getstarted/#install-litmus) - Ensure that the `disk-fill` experiment resource is available in the cluster by executing `kubectl get chaosexperiments` in the desired namespace If not, install from [here](https://hub.litmuschaos.io/api/chaos/1.7.0?file=charts/generic/disk-fill/experiment.yaml) - Cluster must run docker container runtime - Appropriate Ephemeral Storage Requests and Limits should be set for the application before running the experiment. An example specification is shown below: ``` apiVersion: v1 kind: Pod metadata: name: frontend spec: containers: - name: db image: mysql env: - name: MYSQL_ROOT_PASSWORD value: "password" resources: requests: ephemeral-storage: "2Gi" limits: ephemeral-storage: "4Gi" - name: wp image: wordpress resources: requests: ephemeral-storage: "2Gi" limits: ephemeral-storage: "4Gi" ``` ## Entry-Criteria - Application pods are healthy before chaos injection. ## Exit-Criteria - Application pods are healthy post chaos injection. ## Details - Causes Disk Stress by filling up the ephemeral storage of the pod (in the /var/lib/docker/container/{{container_id}}) on any given node. - Causes the application pod to get evicted if the capacity filled exceeds the pod's ephemeral storage limit. - Tests the Ephemeral Storage Limits, to ensure those parameters are sufficient. - Tests the application's resiliency to disk stress/replica evictions. ## Integrations - Disk Fill can be effected using the chaos library: `litmus`, which makes use of `dd` to create a file of specified capacity on the node. - The desired chaoslib can be selected by setting the above options as value for the env variable `LIB` ## Steps to Execute the Chaos Experiment - This Chaos Experiment can be triggered by creating a ChaosEngine resource on the cluster. To understand the values to provide in a ChaosEngine specification, refer [Getting Started](getstarted.md/#prepare-chaosengine) - Follow the steps in the sections below to create the chaosServiceAccount, prepare the ChaosEngine & execute the experiment. ### Prepare chaosServiceAccount - Use this sample RBAC manifest to create a chaosServiceAccount in the desired (app) namespace. This example consists of the minimum necessary role permissions to execute the experiment. #### Sample Rbac Manifest [embedmd]: # "https://raw.githubusercontent.com/litmuschaos/chaos-charts/master/charts/generic/disk-fill/rbac.yaml yaml" ```yaml --- apiVersion: v1 kind: ServiceAccount metadata: name: disk-fill-sa namespace: default labels: name: disk-fill-sa --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRole metadata: name: disk-fill-sa labels: name: disk-fill-sa rules: - apiGroups: ["", "apps", "litmuschaos.io", "batch"] resources: [ "pods", "jobs", "pods/exec", "events", "pods/log", "chaosengines", "chaosexperiments", "chaosresults", ] verbs: ["create", "list", "get", "patch", "update", "delete"] --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: disk-fill-sa labels: name: disk-fill-sa roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: disk-fill-sa subjects: - kind: ServiceAccount name: disk-fill-sa namespace: default ``` ### Prepare ChaosEngine - Provide the application info in `spec.appinfo` - Provide the auxiliary applications info (ns & labels) in `spec.auxiliaryAppInfo` - Override the experiment tunables if desired in `experiments.spec.components.env` - To understand the values to provided in a ChaosEngine specification, refer [ChaosEngine Concepts](chaosengine-concepts.md) #### Supported Experiment Tunables
Variables Description Specify In ChaosEngine Notes
FILL_PERCENTAGE Percentage to fill the Ephemeral storage limit Mandatory Can be set to more than 100 also, to force evict the pod
TARGET_CONTAINER Name of container which is subjected to disk-fill Mandatory
CONTAINER_PATH Storage Location of containers Optional Defaults to '/var/lib/docker/containers'
TOTAL_CHAOS_DURATION The time duration for chaos insertion (sec) Optional Defaults to 60s
LIB The chaos lib used to inject the chaos Optional Defaults to `litmus`
LIB_IMAGE The image used to fill the disk Optional Defaults to `litmuschaos/litmus-go:latest`
RAMP_TIME Period to wait before injection of chaos in sec Optional
INSTANCE_ID A user-defined string that holds metadata/info about current run/instance of chaos. Ex: 04-05-2020-9-00. This string is appended as suffix in the chaosresult CR name. Optional Ensure that the overall length of the chaosresult CR is still < 64 characters
#### Sample ChaosEngine Manifest [embedmd]: # "https://raw.githubusercontent.com/litmuschaos/chaos-charts/master/charts/generic/disk-fill/engine.yaml yaml" ```yaml apiVersion: litmuschaos.io/v1alpha1 kind: ChaosEngine metadata: name: nginx-chaos namespace: default spec: # It can be true/false annotationCheck: "false" # It can be active/stop engineState: "active" #ex. values: ns1:name=percona,ns2:run=nginx auxiliaryAppInfo: "" appinfo: appns: "default" applabel: "app=nginx" appkind: "deployment" chaosServiceAccount: disk-fill-sa monitoring: false # It can be delete/retain jobCleanUpPolicy: "delete" experiments: - name: disk-fill spec: components: env: # specify the fill percentage according to the disk pressure required - name: FILL_PERCENTAGE value: "80" - name: TARGET_CONTAINER value: "nginx" ``` ### Create the ChaosEngine Resource - Create the ChaosEngine manifest prepared in the previous step to trigger the Chaos. `kubectl apply -f chaosengine.yml` - If the chaos experiment is not executed, refer to the [troubleshooting](https://docs.litmuschaos.io/docs/faq-troubleshooting/) section to identify the root cause and fix the issues. ### Watch Chaos progress - View the status of the pods as they are subjected to disk stress. `watch -n 1 kubectl get pods -n ` - Monitor the capacity filled up on the host filesystem `watch -n 1 du -kh /var/lib/docker/containers/` ### Abort/Restart the Chaos Experiment - To stop the pod-delete experiment immediately, either delete the ChaosEngine resource or execute the following command: `kubectl patch chaosengine -n --type merge --patch '{"spec":{"engineState":"stop"}}'` - To restart the experiment, either re-apply the ChaosEngine YAML or execute the following command: `kubectl patch chaosengine -n --type merge --patch '{"spec":{"engineState":"active"}}'` **Notes:** - The abort will stop further fill of the local disk, but doesn't reclaim the used capacity. This is a manual operation as of today. The auto-rollback, i.e., in this case reclaim of currently filled disk-space will be implemented in a future release - However, upon graceful completion of the experiment (i.e.,un-aborted), the space is automatically reclaimed as the chaos impact is reverted. ### Check Chaos Experiment Result - Check whether the application is resilient to the container kill, once the experiment (job) is completed. The ChaosResult resource name is derived like this: `-`. `kubectl describe chaosresult nginx-chaos-disk-fill -n ` ### Disk Fill Experiment Demo - A sample recording of this experiment execution is provided [here](https://www.youtube.com/watch?v=pbok737rUPQ).