--- reviewers: - janetkuo - thockin content_type: concept title: Application Introspection and Debugging --- Once your application is running, you'll inevitably need to debug problems with it. Earlier we described how you can use `kubectl get pods` to retrieve simple status information about your pods. But there are a number of ways to get even more information about your application. ## Using `kubectl describe pod` to fetch details about pods For this example we'll use a Deployment to create two pods, similar to the earlier example. {{< codenew file="application/nginx-with-request.yaml" >}} Create deployment by running following command: ```shell kubectl apply -f https://k8s.io/examples/application/nginx-with-request.yaml ``` ```none deployment.apps/nginx-deployment created ``` Check pod status by following command: ```shell kubectl get pods ``` ```none NAME READY STATUS RESTARTS AGE nginx-deployment-67d4bdd6f5-cx2nz 1/1 Running 0 13s nginx-deployment-67d4bdd6f5-w6kd7 1/1 Running 0 13s ``` We can retrieve a lot more information about each of these pods using `kubectl describe pod`. For example: ```shell kubectl describe pod nginx-deployment-67d4bdd6f5-w6kd7 ``` ```none Name: nginx-deployment-67d4bdd6f5-w6kd7 Namespace: default Priority: 0 Node: kube-worker-1/192.168.0.113 Start Time: Thu, 17 Feb 2022 16:51:01 -0500 Labels: app=nginx pod-template-hash=67d4bdd6f5 Annotations: Status: Running IP: 10.88.0.3 IPs: IP: 10.88.0.3 IP: 2001:db8::1 Controlled By: ReplicaSet/nginx-deployment-67d4bdd6f5 Containers: nginx: Container ID: containerd://5403af59a2b46ee5a23fb0ae4b1e077f7ca5c5fb7af16e1ab21c00e0e616462a Image: nginx Image ID: docker.io/library/nginx@sha256:2834dc507516af02784808c5f48b7cbe38b8ed5d0f4837f16e78d00deb7e7767 Port: 80/TCP Host Port: 0/TCP State: Running Started: Thu, 17 Feb 2022 16:51:05 -0500 Ready: True Restart Count: 0 Limits: cpu: 500m memory: 128Mi Requests: cpu: 500m memory: 128Mi Environment: Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bgsgp (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: kube-api-access-bgsgp: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Guaranteed Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 34s default-scheduler Successfully assigned default/nginx-deployment-67d4bdd6f5-w6kd7 to kube-worker-1 Normal Pulling 31s kubelet Pulling image "nginx" Normal Pulled 30s kubelet Successfully pulled image "nginx" in 1.146417389s Normal Created 30s kubelet Created container nginx Normal Started 30s kubelet Started container nginx ``` Here you can see configuration information about the container(s) and Pod (labels, resource requirements, etc.), as well as status information about the container(s) and Pod (state, readiness, restart count, events, etc.). The container state is one of Waiting, Running, or Terminated. Depending on the state, additional information will be provided -- here you can see that for a container in Running state, the system tells you when the container started. Ready tells you whether the container passed its last readiness probe. (In this case, the container does not have a readiness probe configured; the container is assumed to be ready if no readiness probe is configured.) Restart Count tells you how many times the container has been restarted; this information can be useful for detecting crash loops in containers that are configured with a restart policy of 'always.' Currently the only Condition associated with a Pod is the binary Ready condition, which indicates that the pod is able to service requests and should be added to the load balancing pools of all matching services. Lastly, you see a log of recent events related to your Pod. The system compresses multiple identical events by indicating the first and last time it was seen and the number of times it was seen. "From" indicates the component that is logging the event, "SubobjectPath" tells you which object (e.g. container within the pod) is being referred to, and "Reason" and "Message" tell you what happened. ## Example: debugging Pending Pods A common scenario that you can detect using events is when you've created a Pod that won't fit on any node. For example, the Pod might request more resources than are free on any node, or it might specify a label selector that doesn't match any nodes. Let's say we created the previous Deployment with 5 replicas (instead of 2) and requesting 600 millicores instead of 500, on a four-node cluster where each (virtual) machine has 1 CPU. In that case one of the Pods will not be able to schedule. (Note that because of the cluster addon pods such as fluentd, skydns, etc., that run on each node, if we requested 1000 millicores then none of the Pods would be able to schedule.) ```shell kubectl get pods ``` ```none NAME READY STATUS RESTARTS AGE nginx-deployment-1006230814-6winp 1/1 Running 0 7m nginx-deployment-1006230814-fmgu3 1/1 Running 0 7m nginx-deployment-1370807587-6ekbw 1/1 Running 0 1m nginx-deployment-1370807587-fg172 0/1 Pending 0 1m nginx-deployment-1370807587-fz9sd 0/1 Pending 0 1m ``` To find out why the nginx-deployment-1370807587-fz9sd pod is not running, we can use `kubectl describe pod` on the pending Pod and look at its events: ```shell kubectl describe pod nginx-deployment-1370807587-fz9sd ``` ```none Name: nginx-deployment-1370807587-fz9sd Namespace: default Node: / Labels: app=nginx,pod-template-hash=1370807587 Status: Pending IP: Controllers: ReplicaSet/nginx-deployment-1370807587 Containers: nginx: Image: nginx Port: 80/TCP QoS Tier: memory: Guaranteed cpu: Guaranteed Limits: cpu: 1 memory: 128Mi Requests: cpu: 1 memory: 128Mi Environment Variables: Volumes: default-token-4bcbi: Type: Secret (a volume populated by a Secret) SecretName: default-token-4bcbi Events: FirstSeen LastSeen Count From SubobjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 1m 48s 7 {default-scheduler } Warning FailedScheduling pod (nginx-deployment-1370807587-fz9sd) failed to fit in any node fit failure on node (kubernetes-node-6ta5): Node didn't have enough resource: CPU, requested: 1000, used: 1420, capacity: 2000 fit failure on node (kubernetes-node-wul5): Node didn't have enough resource: CPU, requested: 1000, used: 1100, capacity: 2000 ``` Here you can see the event generated by the scheduler saying that the Pod failed to schedule for reason `FailedScheduling` (and possibly others). The message tells us that there were not enough resources for the Pod on any of the nodes. To correct this situation, you can use `kubectl scale` to update your Deployment to specify four or fewer replicas. (Or you could leave the one Pod pending, which is harmless.) Events such as the ones you saw at the end of `kubectl describe pod` are persisted in etcd and provide high-level information on what is happening in the cluster. To list all events you can use ```shell kubectl get events ``` but you have to remember that events are namespaced. This means that if you're interested in events for some namespaced object (e.g. what happened with Pods in namespace `my-namespace`) you need to explicitly provide a namespace to the command: ```shell kubectl get events --namespace=my-namespace ``` To see events from all namespaces, you can use the `--all-namespaces` argument. In addition to `kubectl describe pod`, another way to get extra information about a pod (beyond what is provided by `kubectl get pod`) is to pass the `-o yaml` output format flag to `kubectl get pod`. This will give you, in YAML format, even more information than `kubectl describe pod`--essentially all of the information the system has about the Pod. Here you will see things like annotations (which are key-value metadata without the label restrictions, that is used internally by Kubernetes system components), restart policy, ports, and volumes. ```shell kubectl get pod nginx-deployment-1006230814-6winp -o yaml ``` ```yaml apiVersion: v1 kind: Pod metadata: creationTimestamp: "2022-02-17T21:51:01Z" generateName: nginx-deployment-67d4bdd6f5- labels: app: nginx pod-template-hash: 67d4bdd6f5 name: nginx-deployment-67d4bdd6f5-w6kd7 namespace: default ownerReferences: - apiVersion: apps/v1 blockOwnerDeletion: true controller: true kind: ReplicaSet name: nginx-deployment-67d4bdd6f5 uid: 7d41dfd4-84c0-4be4-88ab-cedbe626ad82 resourceVersion: "1364" uid: a6501da1-0447-4262-98eb-c03d4002222e spec: containers: - image: nginx imagePullPolicy: Always name: nginx ports: - containerPort: 80 protocol: TCP resources: limits: cpu: 500m memory: 128Mi requests: cpu: 500m memory: 128Mi terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-bgsgp readOnly: true dnsPolicy: ClusterFirst enableServiceLinks: true nodeName: kube-worker-1 preemptionPolicy: PreemptLowerPriority priority: 0 restartPolicy: Always schedulerName: default-scheduler securityContext: {} serviceAccount: default serviceAccountName: default terminationGracePeriodSeconds: 30 tolerations: - effect: NoExecute key: node.kubernetes.io/not-ready operator: Exists tolerationSeconds: 300 - effect: NoExecute key: node.kubernetes.io/unreachable operator: Exists tolerationSeconds: 300 volumes: - name: kube-api-access-bgsgp projected: defaultMode: 420 sources: - serviceAccountToken: expirationSeconds: 3607 path: token - configMap: items: - key: ca.crt path: ca.crt name: kube-root-ca.crt - downwardAPI: items: - fieldRef: apiVersion: v1 fieldPath: metadata.namespace path: namespace status: conditions: - lastProbeTime: null lastTransitionTime: "2022-02-17T21:51:01Z" status: "True" type: Initialized - lastProbeTime: null lastTransitionTime: "2022-02-17T21:51:06Z" status: "True" type: Ready - lastProbeTime: null lastTransitionTime: "2022-02-17T21:51:06Z" status: "True" type: ContainersReady - lastProbeTime: null lastTransitionTime: "2022-02-17T21:51:01Z" status: "True" type: PodScheduled containerStatuses: - containerID: containerd://5403af59a2b46ee5a23fb0ae4b1e077f7ca5c5fb7af16e1ab21c00e0e616462a image: docker.io/library/nginx:latest imageID: docker.io/library/nginx@sha256:2834dc507516af02784808c5f48b7cbe38b8ed5d0f4837f16e78d00deb7e7767 lastState: {} name: nginx ready: true restartCount: 0 started: true state: running: startedAt: "2022-02-17T21:51:05Z" hostIP: 192.168.0.113 phase: Running podIP: 10.88.0.3 podIPs: - ip: 10.88.0.3 - ip: 2001:db8::1 qosClass: Guaranteed startTime: "2022-02-17T21:51:01Z" ``` ## Example: debugging a down/unreachable node Sometimes when debugging it can be useful to look at the status of a node -- for example, because you've noticed strange behavior of a Pod that's running on the node, or to find out why a Pod won't schedule onto the node. As with Pods, you can use `kubectl describe node` and `kubectl get node -o yaml` to retrieve detailed information about nodes. For example, here's what you'll see if a node is down (disconnected from the network, or kubelet dies and won't restart, etc.). Notice the events that show the node is NotReady, and also notice that the pods are no longer running (they are evicted after five minutes of NotReady status). ```shell kubectl get nodes ``` ```none NAME STATUS ROLES AGE VERSION kube-worker-1 NotReady 1h v1.23.3 kubernetes-node-bols Ready 1h v1.23.3 kubernetes-node-st6x Ready 1h v1.23.3 kubernetes-node-unaj Ready 1h v1.23.3 ``` ```shell kubectl describe node kube-worker-1 ``` ```none Name: kube-worker-1 Roles: Labels: beta.kubernetes.io/arch=amd64 beta.kubernetes.io/os=linux kubernetes.io/arch=amd64 kubernetes.io/hostname=kube-worker-1 kubernetes.io/os=linux Annotations: kubeadm.alpha.kubernetes.io/cri-socket: /run/containerd/containerd.sock node.alpha.kubernetes.io/ttl: 0 volumes.kubernetes.io/controller-managed-attach-detach: true CreationTimestamp: Thu, 17 Feb 2022 16:46:30 -0500 Taints: node.kubernetes.io/unreachable:NoExecute node.kubernetes.io/unreachable:NoSchedule Unschedulable: false Lease: HolderIdentity: kube-worker-1 AcquireTime: RenewTime: Thu, 17 Feb 2022 17:13:09 -0500 Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- NetworkUnavailable False Thu, 17 Feb 2022 17:09:13 -0500 Thu, 17 Feb 2022 17:09:13 -0500 WeaveIsUp Weave pod has set this MemoryPressure Unknown Thu, 17 Feb 2022 17:12:40 -0500 Thu, 17 Feb 2022 17:13:52 -0500 NodeStatusUnknown Kubelet stopped posting node status. DiskPressure Unknown Thu, 17 Feb 2022 17:12:40 -0500 Thu, 17 Feb 2022 17:13:52 -0500 NodeStatusUnknown Kubelet stopped posting node status. PIDPressure Unknown Thu, 17 Feb 2022 17:12:40 -0500 Thu, 17 Feb 2022 17:13:52 -0500 NodeStatusUnknown Kubelet stopped posting node status. Ready Unknown Thu, 17 Feb 2022 17:12:40 -0500 Thu, 17 Feb 2022 17:13:52 -0500 NodeStatusUnknown Kubelet stopped posting node status. Addresses: InternalIP: 192.168.0.113 Hostname: kube-worker-1 Capacity: cpu: 2 ephemeral-storage: 15372232Ki hugepages-2Mi: 0 memory: 2025188Ki pods: 110 Allocatable: cpu: 2 ephemeral-storage: 14167048988 hugepages-2Mi: 0 memory: 1922788Ki pods: 110 System Info: Machine ID: 9384e2927f544209b5d7b67474bbf92b System UUID: aa829ca9-73d7-064d-9019-df07404ad448 Boot ID: 5a295a03-aaca-4340-af20-1327fa5dab5c Kernel Version: 5.13.0-28-generic OS Image: Ubuntu 21.10 Operating System: linux Architecture: amd64 Container Runtime Version: containerd://1.5.9 Kubelet Version: v1.23.3 Kube-Proxy Version: v1.23.3 Non-terminated Pods: (4 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age --------- ---- ------------ ---------- --------------- ------------- --- default nginx-deployment-67d4bdd6f5-cx2nz 500m (25%) 500m (25%) 128Mi (6%) 128Mi (6%) 23m default nginx-deployment-67d4bdd6f5-w6kd7 500m (25%) 500m (25%) 128Mi (6%) 128Mi (6%) 23m kube-system kube-proxy-dnxbz 0 (0%) 0 (0%) 0 (0%) 0 (0%) 28m kube-system weave-net-gjxxp 100m (5%) 0 (0%) 200Mi (10%) 0 (0%) 28m Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits -------- -------- ------ cpu 1100m (55%) 1 (50%) memory 456Mi (24%) 256Mi (13%) ephemeral-storage 0 (0%) 0 (0%) hugepages-2Mi 0 (0%) 0 (0%) Events: ... ``` ```shell kubectl get node kube-worker-1 -o yaml ``` ```yaml apiVersion: v1 kind: Node metadata: annotations: kubeadm.alpha.kubernetes.io/cri-socket: /run/containerd/containerd.sock node.alpha.kubernetes.io/ttl: "0" volumes.kubernetes.io/controller-managed-attach-detach: "true" creationTimestamp: "2022-02-17T21:46:30Z" labels: beta.kubernetes.io/arch: amd64 beta.kubernetes.io/os: linux kubernetes.io/arch: amd64 kubernetes.io/hostname: kube-worker-1 kubernetes.io/os: linux name: kube-worker-1 resourceVersion: "4026" uid: 98efe7cb-2978-4a0b-842a-1a7bf12c05f8 spec: {} status: addresses: - address: 192.168.0.113 type: InternalIP - address: kube-worker-1 type: Hostname allocatable: cpu: "2" ephemeral-storage: "14167048988" hugepages-2Mi: "0" memory: 1922788Ki pods: "110" capacity: cpu: "2" ephemeral-storage: 15372232Ki hugepages-2Mi: "0" memory: 2025188Ki pods: "110" conditions: - lastHeartbeatTime: "2022-02-17T22:20:32Z" lastTransitionTime: "2022-02-17T22:20:32Z" message: Weave pod has set this reason: WeaveIsUp status: "False" type: NetworkUnavailable - lastHeartbeatTime: "2022-02-17T22:20:15Z" lastTransitionTime: "2022-02-17T22:13:25Z" message: kubelet has sufficient memory available reason: KubeletHasSufficientMemory status: "False" type: MemoryPressure - lastHeartbeatTime: "2022-02-17T22:20:15Z" lastTransitionTime: "2022-02-17T22:13:25Z" message: kubelet has no disk pressure reason: KubeletHasNoDiskPressure status: "False" type: DiskPressure - lastHeartbeatTime: "2022-02-17T22:20:15Z" lastTransitionTime: "2022-02-17T22:13:25Z" message: kubelet has sufficient PID available reason: KubeletHasSufficientPID status: "False" type: PIDPressure - lastHeartbeatTime: "2022-02-17T22:20:15Z" lastTransitionTime: "2022-02-17T22:15:15Z" message: kubelet is posting ready status. AppArmor enabled reason: KubeletReady status: "True" type: Ready daemonEndpoints: kubeletEndpoint: Port: 10250 nodeInfo: architecture: amd64 bootID: 22333234-7a6b-44d4-9ce1-67e31dc7e369 containerRuntimeVersion: containerd://1.5.9 kernelVersion: 5.13.0-28-generic kubeProxyVersion: v1.23.3 kubeletVersion: v1.23.3 machineID: 9384e2927f544209b5d7b67474bbf92b operatingSystem: linux osImage: Ubuntu 21.10 systemUUID: aa829ca9-73d7-064d-9019-df07404ad448 ``` ## {{% heading "whatsnext" %}} Learn about additional debugging tools, including: * [Logging](/docs/concepts/cluster-administration/logging/) * [Monitoring](/docs/tasks/debug-application-cluster/resource-usage-monitoring/) * [Getting into containers via `exec`](/docs/tasks/debug-application-cluster/get-shell-running-container/) * [Connecting to containers via proxies](/docs/tasks/extend-kubernetes/http-proxy-access-api/) * [Connecting to containers via port forwarding](/docs/tasks/access-application-cluster/port-forward-access-application-cluster/) * [Inspect Kubernetes node with crictl](/docs/tasks/debug-application-cluster/crictl/)