diff --git a/contributors/devel/sig-scheduling/scheduler_queues.md b/contributors/devel/sig-scheduling/scheduler_queues.md new file mode 100644 index 000000000..e4eff5fb5 --- /dev/null +++ b/contributors/devel/sig-scheduling/scheduler_queues.md @@ -0,0 +1,99 @@ +# Scheduling queue in kube-scheduler + +Queueing mechanism is an integral part of the scheduler. It allows the scheduler +to pick the most suitable pod for the next scheduling cycle. Given a pod can +specify various conditions that have to be met at the time of scheduling, +such as existence of a persistent volume, compliance with pod anti-affinity rules +or toleration of node taints, the mechanism needs to be able to postpone +the scheduling action until the cluster may meet all the conditions for +the successful scheduling. The mechanism relies on three queues: +- active ([activeQ](https://github.com/kubernetes/kubernetes/blob/4cc1127e9251fff364d5c77e2a9a9c3ad42383ab/pkg/scheduler/internal/queue/scheduling_queue.go#L130)): providing pods for immediate scheduling +- unschedulable ([unschedulableQ](https://github.com/kubernetes/kubernetes/blob/4cc1127e9251fff364d5c77e2a9a9c3ad42383ab/pkg/scheduler/internal/queue/scheduling_queue.go#L135)): for parking pods which are waiting for certain condition(s) to happen +- backoff ([podBackoffQ](https://github.com/kubernetes/kubernetes/blob/4cc1127e9251fff364d5c77e2a9a9c3ad42383ab/pkg/scheduler/internal/queue/scheduling_queue.go#L133)): exponentially postponing pods which failed + to be scheduled (e.g. volume still getting created) but are expected to get scheduled eventually. + +In addition, the scheduling queue mechanism has two periodical flushing goroutines +running in the background responsible for moving pods to the active queue: +- [flushUnschedulableQLeftover](https://github.com/kubernetes/kubernetes/blob/4cc1127e9251fff364d5c77e2a9a9c3ad42383ab/pkg/scheduler/internal/queue/scheduling_queue.go#L350): running every 30 seconds moving pods from unschedulable + queue to allow unschedulable pods that were not moved by any event + to be retried again. Pod has to stay for at least 30 seconds in the queue to get moved. + In the worst case it can take up to 60 seconds to have a pod moved. +- [flushBackoffQCompleted](https://github.com/kubernetes/kubernetes/blob/4cc1127e9251fff364d5c77e2a9a9c3ad42383ab/pkg/scheduler/internal/queue/scheduling_queue.go#L324): running every second moving pods that were backed off + long enough to the active queue. + +Both retry periods for the goroutines are fixed and non-configurable. +Also, in response to certain events, the scheduler +move pods from either queue to the active queue (by invoking [MoveAllToActiveOrBackoffQueue](https://github.com/kubernetes/kubernetes/blob/4cc1127e9251fff364d5c77e2a9a9c3ad42383ab/pkg/scheduler/internal/queue/scheduling_queue.go#L493)). +Example events include a node addition or update, an existing pod being deleted etc. + +![Pods moving between queues](scheduling_queues.png "Pods moving between queues") + +## Active queue (heap) + +A queue with the highest priority pod at the top by default. The ordering +can be customized via QueueSort extension point. Newly created pods, with empty `.spec.nodeName`, +are added to the queue as they come. In each scheduling cycle the scheduler takes +one pod from the queue and tries to schedule it. In case the scheduling algorithm +fails (e.g. plugins error, binding error), the pod is moved to the unschedulable queue. +Or, moved to the backoff queue if a move request was issued at the same or newer time. +The move request signals a move of pods from unschedulable to active, respectively backoff queue. +If a pod is scheduled without an error, it is removed from all queues. + +## Backoff queue (heap) +Queue keeping pods in a waiting state to avoid continuous retries. Queue ordering +keeps a pod with the shortest backoff timeout at the top. The more times a pod gets +backed off, the longer it takes for the pod to re-enter the active queue. The backoff +timeout grows exponentially with each failed scheduling attempt until it reaches its maximum. +Scheduler allows to configure initial backoff (set to 1 second by default) and maximum +backoff (set to 10 seconds by default). A pod can get to the backoff queue +when a move request (see below) is issued. + +As an example a pod with 3 failed attempts gets the target backoff timeout +set to curTime + 2s^3 (8s). With 5 failed attempts the timeout gets set to curTime +2s^5 (32s). +In case the maximum backoff is too low (e.g. the default 10s), a pod can get to the active +queue too often. So it’s recommended to configure the maximum backoff to fit the workloads +so the pods stay in the backoff queue long enough to avoid flooding the active queue +with pods failing too often to be scheduled. + +## Unschedulable queue (map) +Queue keeping all pods that failed to be scheduled and were not subject to a move request. +Pods are kept in the queue until a move request is issued. + +## Moving request + +Moving request triggers an event responsible for moving pods from +unschedulable queue to either the active or the backoff queue. Different cluster +events can asynchronously trigger a moving request and make unschedulable +pods (that were tried before) schedulable again. The events currently include +changes in pods, nodes, services, PVs, PVCs, storage classes and CSI nodes. + +It’s possible that a pod fails to be scheduled while a moving request gets issued. +Due to this event, the pod might now be schedulable and the following mechanism +allows such pod to be retried. Every moving request operation stores the current +scheduling cycle under [moveRequestCycle](https://github.com/kubernetes/kubernetes/blob/4cc1127e9251fff364d5c77e2a9a9c3ad42383ab/pkg/scheduler/internal/queue/scheduling_queue.go#L523) variable. After a pod fails scheduling, +it is regularly put in the unschedulable queue. Unless moveRequestCycle +is the current scheduling cycle, in which case the pod takes a shortcut +and gets moved right under the backoff queue. + +**Examples**: +- When a pod is scheduled, some pods in the unschedulable queue with matching + affinity can be made schedulable. If matching affinity is the only required + condition for scheduling, issuing a moving request for those pods will allow + them to get finally scheduled. +- A pod is getting processed by filter plugins which give no nodes left for scheduling. + Meantime an asynchronous moving request gets issued as a reaction on a new node event. + Moving the pod under the backoff queue will allow the pod to be moved sooner + into the active queue and check if the new node is eligible for scheduling. + +## Metrics + +The scheduling queue populates two metrics: +[pending_pods](https://github.com/kubernetes/kubernetes/blob/4cc1127e9251fff364d5c77e2a9a9c3ad42383ab/pkg/scheduler/metrics/metrics.go#L83-L89) and +[queue_incoming_pods_total](https://github.com/kubernetes/kubernetes/blob/4cc1127e9251fff364d5c77e2a9a9c3ad42383ab/pkg/scheduler/metrics/metrics.go#L141-L147). +All three queues count how many pods are pending in each queue and how many +times a pod was enqueued into each queue. Including which event was responsible +for the enqueueing. The events can include failed scheduling attempts, +pod finishing backoff, node added, service updated, etc. The metrics allow us +to see how many pods are present in each queue. Allowing to see how often pods +are unschedulable, what’s the scheduler throughput, or which events are moving +the pods from one queue to another most often. diff --git a/contributors/devel/sig-scheduling/scheduling_queues.png b/contributors/devel/sig-scheduling/scheduling_queues.png new file mode 100644 index 000000000..1ffcc77ae Binary files /dev/null and b/contributors/devel/sig-scheduling/scheduling_queues.png differ