litmuschaos
/
litmus
mirror of https://github.com/litmuschaos/litmus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
litmus/monitoring/utils/sample-application-under-test/sock-shop/README.md

4.6 KiB
Raw Blame History

Demonstration

Monitor Chaos on Sock-Shop

Run chaos experiments and workflows on sock-shop application with grafana dashboard to monitor it.

Setup Sock-Shop Microservices Application

  • Apply the sock-shop microservices manifests

    kubectl apply -f .

  • Wait until all services are up. Verify via kubectl get pods -n sock-shop

Setup the Monitoring Components

  • create service monitors for all the application services if using prometheus operator with service monitors.

    kubectl -n sock-shop apply -f ../../sample-application-service-monitors/sock-shop/

Import the grafana dashboard

image

  • Import the grafana dashboard "Sock-Shop Performance" provided here

Execute the Chaos Experiments

  • For the sake of illustration, let us execute node and pod level, CPU hog experiments on the catalogue microservice & Memory Hog experiments on the orders microservice in a staggered manner.

    kubectl apply -f ../../sample-chaos-injectors/chaos-experiments/catalogue/catalogue-pod-cpu-hog.yaml

    Wait for ~60s

    kubectl apply -f ../../sample-chaos-injectors/chaos-experiments/orders/orders-pod-memory-hog.yaml

    Wait for ~60s

    kubectl apply -f ../../sample-chaos-injectors/chaos-experiments/catalogue/catalogue-node-cpu-hog.yaml

    Wait for ~60s

    kubectl apply -f ../../sample-chaos-injectors/chaos-experiments/orders/orders-node-memory-hog.yaml

  • Verify execution of chaos experiments

    kubectl describe chaosengine catalogue-pod-cpu-hog -n litmus
kubectl describe chaosengine orders-pod-memory-hog -n litmus
kubectl describe chaosengine catalogue-node-cpu-hog -n litmus
kubectl describe chaosengine orders-node-memory-hog -n litmus

Visualize Chaos Impact

  • Observe the impact of chaos injection through increased Latency & reduced QPS (queries per second) on the microservices under test.

    image

    image

Inject continous chaos using Argo CD.

  • Install Chaos workflow infrastructure.

    • Create argo namespace

      kubectl create ns argo

    • Create the CRDs, workflow controller deployment with associated RBAC.

      kubectl apply -f https://raw.githubusercontent.com/argoproj/argo/stable/manifests/install.yaml -n argo

    • Install the argo CLI on the test harness machine (where the kubeconfig is available)

      # Download the binary
curl -sLO https://github.com/argoproj/argo/releases/download/v2.11.0/argo-linux-amd64.gz

# Unzip
gunzip argo-linux-amd64.gz

# Make binary executable
chmod +x argo-linux-amd64

# Move binary to path
mv ./argo-linux-amd64 /usr/local/bin/argo

# Test installation
argo version

  • Create the Argo Access ServiceAccount

    kubectl apply -f https://raw.githubusercontent.com/litmuschaos/chaos-workflows/master/Argo/argo-access.yaml -n litmus

  • Run one or more of the litmuschaos experiments as Chaos workflows using argo CLI or kubectl.

    Node CPU hog

    argo cron create ../../sample-chaos-injectors/chaos-workflows-with-argo-CD/catalogue/catalogue-node-cpu-hog-workflow.yaml -n litmus

    Node memory hog

    argo cron create ../../sample-chaos-injectors/chaos-workflows-with-argo-CD/orders/orders-node-memory-hog-workflow.yaml -n litmus

    Pod CPU hog

    kubectl apply -f ../../sample-chaos-injectors/chaos-workflows-with-argo-CD/catalogue/catalogue-pod-cpu-hog-workflow.yaml -n litmus

    Pod memory hog

    kubectl apply -f ../../sample-chaos-injectors/chaos-workflows-with-argo-CD/orders/orders-pod-memory-hog-workflow.yaml -n litmus

  • Visualize the Chaos cron workflow through argo UI by obtaining Node port or Load Balancer IP.

    kubectl patch svc argo-server -n argo -p '{"spec": {"type": "NodePort"}}'

    OR

    kubectl patch svc argo-server -n argo -p '{"spec": {"type": "LoadBalancer"}}'

    image

    image