litmuschaos
/
litmus-docs
mirror of https://github.com/litmuschaos/litmus-docs.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
litmus-docs/website/docs/pod-network-duplication.md

10 KiB
Raw Blame History

id title sidebar_label
pod-network-duplication Pod Network Duplication Experiment Details Pod Network Duplication

Experiment Metadata

Type Description Tested K8s Platform
Generic Inject Packet Duplication Into Application Pod GKE, Packet(Kubeadm), EKS, Minikube > v1.6.0, AKS

Prerequisites

  • Ensure that the Litmus Chaos Operator is running by executing kubectl get pods in operator namespace (typically, litmus). If not, install from here
  • Ensure that the pod-network-duplication experiment resource is available in the cluster by executing kubectl get chaosexperiments in the desired namespace. If not, install from here

Entry Criteria

  • Application pods are healthy before chaos injection

Exit Criteria

  • Application pods are healthy post chaos injection

Details

  • pod-network-duplication injects chaos to disrupt network connectivity to kubernetes pods.
  • The application pod should be healthy once chaos is stopped. Service-requests should be served despite chaos.
  • Causes Injection of network duplication on the specified container by starting a traffic control (tc) process with netem rules to add egress delays. It Can test the application's resilience to duplicate network.

Steps to Execute the Chaos Experiment

  • This Chaos Experiment can be triggered by creating a ChaosEngine resource on the cluster. To understand the values to provide in a ChaosEngine specification, refer Getting Started

  • Follow the steps in the sections below to create the chaosServiceAccount, prepare the ChaosEngine & execute the experiment.

Prepare chaosServiceAccount

  • Use this sample RBAC manifest to create a chaosServiceAccount in the desired (app) namespace. This example consists of the minimum necessary role permissions to execute the experiment.

Sample Rbac Manifest

apiVersion: v1
kind: ServiceAccount
metadata:
  name: pod-network-duplication-sa
  namespace: default
  labels:
    name: pod-network-duplication-sa
    app.kubernetes.io/part-of: litmus
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-network-duplication-sa
  namespace: default
  labels:
    name: pod-network-duplication-sa
    app.kubernetes.io/part-of: litmus
rules:
  - apiGroups: ["", "litmuschaos.io", "batch"]
    resources:
      [
        "pods",
        "jobs",
        "events",
        "pods/log",
        "pods/exec",
        "chaosengines",
        "chaosexperiments",
        "chaosresults",
      ]
    verbs:
      ["create", "list", "get", "patch", "update", "delete", "deletecollection"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: pod-network-duplication-sa
  namespace: default
  labels:
    name: pod-network-duplication-sa
    app.kubernetes.io/part-of: litmus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: pod-network-duplication-sa
subjects:
  - kind: ServiceAccount
    name: pod-network-duplication-sa
    namespace: default

Note: In case of restricted systems/setup, create a PodSecurityPolicy(psp) with the required permissions. The chaosServiceAccount can subscribe to work around the respective limitations. An example of a standard psp that can be used for litmus chaos experiments can be found here.

Prepare ChaosEngine

  • Provide the application info in spec.appinfo
  • Override the experiment tunables if desired in experiments.spec.components.env
  • To understand the values to provided in a ChaosEngine specification, refer ChaosEngine Concepts

Supported Experiment Tunables

Variables Description Specify In ChaosEngine Notes
NETWORK_INTERFACE Name of ethernet interface considered for shaping traffic Mandatory
TARGET_CONTAINER Name of container which is subjected to network latency Optional By default it will take the first container of the target application pod
NETWORK_PACKET_DUPLICATION_PERCENTAGE The packet duplication in percentage Optional Default to 100 percentage
TOTAL_CHAOS_DURATION The time duration for chaos insertion (seconds) Optional Default (60s)
PODS_AFFECTED_PERC The Percentage of total pods to target Optional Defaults to 0 (corresponds to 1 replica), provide numeric value only
CONTAINER_RUNTIME container runtime interface for the cluster Optional Defaults to docker, supported values: docker, containerd and crio for litmus and only docker for pumba LIB
SOCKET_PATH Path of the containerd/crio/docker socket file Optional Defaults to `/var/run/docker.sock`
TARGET_PODS Comma separated list of application pod name subjected to pod network duplication chaos Optional If not provided, it will select target pods randomly based on provided appLabels
DESTINATION_IPS IP addresses of the services or pods, the accessibility to which, is impacted Optional if not provided, it will induce network chaos for all ips/destinations
DESTINATION_HOSTS DNS Names/FQDN names of the services, the accessibility to which, is impacted Optional if not provided, it will induce network chaos for all ips/destinations or DESTINATION_IPS if already defined
LIB The chaos lib used to inject the chaos Optional Default value: litmus, supported values: pumba and litmus
TC_IMAGE Image used for traffic control in linux Optional default value is `gaiadocker/iproute2`
LIB_IMAGE Image used to run the netem command Optional Defaults to `litmuschaos/go-runner:latest`
RAMP_TIME Period to wait before and after injection of chaos in sec Optional
SEQUENCE It defines sequence of chaos execution for multiple target pods Optional Default value: parallel. Supported: serial, parallel
INSTANCE_ID A user-defined string that holds metadata/info about current run/instance of chaos. Ex: 04-05-2020-9-00. This string is appended as suffix in the chaosresult CR name. Optional Ensure that the overall length of the chaosresult CR is still < 64 characters

Sample ChaosEngine Manifest

# chaosengine.yaml
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: nginx-network-chaos
  namespace: default
spec:
  # It can be delete/retain
  jobCleanUpPolicy: "delete"
  # It can be true/false
  annotationCheck: "true"
  # It can be active/stop
  engineState: "active"
  monitoring: false
  appinfo:
    appns: "default"
    # FYI, To see app label, apply kubectl get pods --show-labels
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: pod-network-duplication-sa
  experiments:
    - name: pod-network-duplication
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: "60" # in seconds

            #Network interface inside target container
            - name: NETWORK_INTERFACE
              value: "eth0"

            - name: NETWORK_PACKET_DUPLICATION_PERCENTAGE
              value: "100"

            # provide the name of container runtime
            # for litmus LIB, it supports docker, containerd, crio
            # for pumba LIB, it supports docker only
            - name: CONTAINER_RUNTIME
              value: "docker"

            # provide the socket file path
            - name: SOCKET_PATH
              value: "/var/run/docker.sock"

Create the ChaosEngine Resource

  • Create the ChaosEngine manifest prepared in the previous step to trigger the Chaos.

    kubectl apply -f chaosengine.yml

  • If the chaos experiment is not executed, refer to the troubleshooting section to identify the root cause and fix the issues.

Watch Chaos progress

  • View network latency by setting up a ping on the affected pod from the cluster nodes

    ping <pod_ip_address>

Abort/Restart the Chaos Experiment

  • To stop the pod-network-duplication experiment immediately, either delete the ChaosEngine resource or execute the following command:

    kubectl patch chaosengine <chaosengine-name> -n <namespace> --type merge --patch '{"spec":{"engineState":"stop"}}'

  • To restart the experiment, either re-apply the ChaosEngine YAML or execute the following command:

    kubectl patch chaosengine <chaosengine-name> -n <namespace> --type merge --patch '{"spec":{"engineState":"active"}'

Check Chaos Experiment Result

  • Check whether the application is resilient to the Pod Network Duplication, once the experiment (job) is completed. The ChaosResult resource name is derived like this: <ChaosEngine-Name>-<ChaosExperiment-Name>.

    kubectl describe chaosresult <ChaosEngine-Name>-<ChaosExperiment-Name> -n <application-namespace>

Application Pod Network Duplication Demo

  • A sample recording of this experiment execution will be provided very soon.