diff --git a/content/en/docs/concepts/policy/limit-range.md b/content/en/docs/concepts/policy/limit-range.md index 455f51b1aa..6fb7b1dcd6 100644 --- a/content/en/docs/concepts/policy/limit-range.md +++ b/content/en/docs/concepts/policy/limit-range.md @@ -1,382 +1,382 @@ ---- -reviewers: -- nelvadas -title: Limit Ranges -content_template: templates/concept -weight: 10 ---- - -{{% capture overview %}} - -By default, containers run with unbounded [compute resources](/docs/user-guide/compute-resources) on a Kubernetes cluster. -With Resource quotas, cluster administrators can restrict the resource consumption and creation on a namespace basis. -Within a namespace, a Pod or Container can consume as much CPU and memory as defined by the namespace's resource quota. There is a concern that one Pod or Container could monopolize all of the resources. Limit Range is a policy to constrain resource by Pod or Container in a namespace. - -{{% /capture %}} - - -{{% capture body %}} - -A limit range, defined by a `LimitRange` object, provides constraints that can: - -- Enforce minimum and maximum compute resources usage per Pod or Container in a namespace. -- Enforce minimum and maximum storage request per PersistentVolumeClaim in a namespace. -- Enforce a ratio between request and limit for a resource in a namespace. -- Set default request/limit for compute resources in a namespace and automatically inject them to Containers at runtime. - -## Enabling Limit Range - -Limit Range support is enabled by default for many Kubernetes distributions. It is -enabled when the apiserver `--enable-admission-plugins=` flag has `LimitRanger` admission controller as -one of its arguments. - -A limit range is enforced in a particular namespace when there is a -`LimitRange` object in that namespace. - -### Overview of Limit Range: - -- The administrator creates one `LimitRange` in one namespace. -- Users create resources like Pods, Containers, and PersistentVolumeClaims in the namespace. -- The `LimitRanger` admission controller enforces defaults limits for all Pods and Container that do not set compute resource requirements and tracks usage to ensure it does not exceed resource minimum , maximum and ratio defined in any `LimitRange` present in the namespace. -- If creating or updating a resource (Pod, Container, PersistentVolumeClaim) violates a limit range constraint, the request to the API server will fail with HTTP status code `403 FORBIDDEN` and a message explaining the constraint that would have been violated. -- If limit range is activated in a namespace for compute resources like `cpu` and `memory`, users must specify - requests or limits for those values; otherwise, the system may reject pod creation. -- LimitRange validations occurs only at Pod Admission stage, not on Running pods. - - -Examples of policies that could be created using limit range are: - -- In a 2 node cluster with a capacity of 8 GiB RAM, and 16 cores, constrain Pods in a namespace to request 100m and not exceeds 500m for CPU , request 200Mi and not exceed 600Mi -- Define default CPU limits and request to 150m and Memory default request to 300Mi for containers started with no cpu and memory requests in their spec. - -In the case where the total limits of the namespace is less than the sum of the limits of the Pods/Containers, -there may be contention for resources; The Containers or Pods will not be created. - -Neither contention nor changes to limitrange will affect already created resources. - -## Limiting Container compute resources - -The following section discusses the creation of a LimitRange acting at Container Level. -A Pod with 04 containers is first created; each container within the Pod has a specific `spec.resource` configuration -each containerwithin the pod is handled differently by the LimitRanger admission controller. - - Create a namespace `limitrange-demo` using the following kubectl command - -```shell -kubectl create namespace limitrange-demo -``` - -To avoid passing the target limitrange-demo in your kubectl commands, change your context with the following command - -```shell -kubectl config set-context --current --namespace=limitrange-demo -``` - -Here is the configuration file for a LimitRange object: -{{< codenew file="admin/resource/limit-mem-cpu-container.yaml" >}} - -This object defines minimum and maximum Memory/CPU limits, default cpu/Memory requests and default limits for CPU/Memory resources to be apply to containers. - -Create the `limit-mem-cpu-per-container` LimitRange in the `limitrange-demo` namespace with the following kubectl command. -```shell -kubectl create -f https://k8s.io/examples/admin/resource/limit-mem-cpu-container.yaml -n limitrange-demo -``` - - -```shell - kubectl describe limitrange/limit-mem-cpu-per-container -n limitrange-demo - ``` - - -```shell -Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio ----- -------- --- --- --------------- ------------- ----------------------- -Container cpu 100m 800m 110m 700m - -Container memory 99Mi 1Gi 111Mi 900Mi - -``` - - - -Here is the configuration file for a Pod with 04 containers to demonstrate LimitRange features : -{{< codenew file="admin/resource/limit-range-pod-1.yaml" >}} - -Create the `busybox1` Pod : - -```shell -kubectl apply -f https://k8s.io/examples/admin/resource/limit-range-pod-1.yaml -n limitrange-demo -``` - -### Container spec with valid CPU/Memory requests and limits -View the the `busybox-cnt01` resource configuration - -```shell -kubectl get po/busybox1 -n limitrange-demo -o json | jq ".spec.containers[0].resources" -``` - -```json -{ - "limits": { - "cpu": "500m", - "memory": "200Mi" - }, - "requests": { - "cpu": "100m", - "memory": "100Mi" - } -} -``` - -- The `busybox-cnt01` Container inside `busybox` Pod defined `requests.cpu=100m` and `requests.memory=100Mi`. -- `100m <= 500m <= 800m` , The container cpu limit (500m) falls inside the authorized CPU limit range. -- `99Mi <= 200Mi <= 1Gi` , The container memory limit (200Mi) falls inside the authorized Memory limit range. -- No request/limits ratio validation for CPU/Memory , thus the container is valid and created. - - -### Container spec with a valid CPU/Memory requests but no limits - -View the `busybox-cnt02` resource configuration - -```shell -kubectl get po/busybox1 -n limitrange-demo -o json | jq ".spec.containers[1].resources" -``` - -```json -{ - "limits": { - "cpu": "700m", - "memory": "900Mi" - }, - "requests": { - "cpu": "100m", - "memory": "100Mi" - } -} -``` -- The `busybox-cnt02` Container inside `busybox1` Pod defined `requests.cpu=100m` and `requests.memory=100Mi` but not limits for cpu and memory. -- The container do not have a limits section, the default limits defined in the limit-mem-cpu-per-container LimitRange object are injected to this container `limits.cpu=700mi` and `limits.memory=900Mi`. -- `100m <= 700m <= 800m` , The container cpu limit (700m) falls inside the authorized CPU limit range. -- `99Mi <= 900Mi <= 1Gi` , The container memory limit (900Mi) falls inside the authorized Memory limit range. -- No request/limits ratio set , thus the container is valid and created. - - -### Container spec with a valid CPU/Memory limits but no requests -View the `busybox-cnt03` resource configuration - -```shell -kubectl get po/busybox1 -n limitrange-demo -o json | jq ".spec.containers[2].resources" -``` -```json -{ - "limits": { - "cpu": "500m", - "memory": "200Mi" - }, - "requests": { - "cpu": "500m", - "memory": "200Mi" - } -} -``` - -- The `busybox-cnt03` Container inside `busybox1` Pod defined `limits.cpu=500m` and `limits.memory=200Mi` but no `requests` for cpu and memory. -- The container do not define a request section, the defaultRequest defined in the limit-mem-cpu-per-container LimitRange is not used to fill its limits section but the limits defined by the container are set as requests `limits.cpu=500m` and `limits.memory=200Mi`. -- `100m <= 500m <= 800m` , The container cpu limit (500m) falls inside the authorized CPU limit range. -- `99Mi <= 200Mi <= 1Gi` , The container memory limit (200Mi) falls inside the authorized Memory limit range. -- No request/limits ratio set , thus the container is valid and created. - - - -### Container spec with no CPU/Memory requests/limits -View the `busybox-cnt04` resource configuration -```shell -kubectl get po/busybox1 -n limitrange-demo -o json | jq ".spec.containers[3].resources" -``` -```json -{ - "limits": { - "cpu": "700m", - "memory": "900Mi" - }, - "requests": { - "cpu": "110m", - "memory": "111Mi" - } -} -``` - -- The `busybox-cnt04` Container inside `busybox1` define neither `limits` nor `requests`. -- The container do not define a limit section, the default limit defined in the limit-mem-cpu-per-container LimitRange is used to fill its request - `limits.cpu=700m and` `limits.memory=900Mi` . -- The container do not define a request section, the defaultRequest defined in the limit-mem-cpu-per-container LimitRange is used to fill its request section requests.cpu=110m and requests.memory=111Mi -- `100m <= 700m <= 800m` , The container cpu limit (700m) falls inside the authorized CPU limit range. -- `99Mi <= 900Mi <= 1Gi` , The container memory limit (900Mi) falls inside the authorized Memory limitrange . -- No request/limits ratio set , thus the container is valid and created. - -All containers defined in the `busybox` Pod passed LimitRange validations, this the Pod is valid and create in the namespace. - -## Limiting Pod compute resources -The following section discusses how to constrain resources at Pod level. - -{{< codenew file="admin/resource/limit-mem-cpu-pod.yaml" >}} - -Without having to delete `busybox1` Pod, create the `limit-mem-cpu-pod` LimitRange in the `limitrange-demo` namespace -```shell -kubectl apply -f https://k8s.io/examples/admin/resource/limit-mem-cpu-pod.yaml -n limitrange-demo -``` -The limitrange is created and limits CPU to 2 Core and Memory to 2Gi per Pod. -```shell -limitrange/limit-mem-cpu-per-pod created -``` -Describe the `limit-mem-cpu-per-pod` limit object using the following kubectl command -```shell -kubectl describe limitrange/limit-mem-cpu-per-pod -``` - -```shell -Name: limit-mem-cpu-per-pod -Namespace: limitrange-demo -Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio ----- -------- --- --- --------------- ------------- ----------------------- -Pod cpu - 2 - - - -Pod memory - 2Gi - - - -``` -Now create the `busybox2` Pod. - -{{< codenew file="admin/resource/limit-range-pod-2.yaml" >}} - -```shell -kubectl apply -f https://k8s.io/examples/admin/resource/limit-range-pod-2.yaml -n limitrange-demo -``` -The `busybox2` Pod definition is identical to `busybox1` but an error is reported since Pod's resources are now limited -```shell -Error from server (Forbidden): error when creating "limit-range-pod-2.yaml": pods "busybox2" is forbidden: [maximum cpu usage per Pod is 2, but limit is 2400m., maximum memory usage per Pod is 2Gi, but limit is 2306867200.] -``` - -```shell -kubectl get po/busybox1 -n limitrange-demo -o json | jq ".spec.containers[].resources.limits.memory" -"200Mi" -"900Mi" -"200Mi" -"900Mi" -``` -`busybox2` Pod will not be admitted on the cluster since the total memory limit of its container is greater than the limit defined in the LimitRange. -`busybox1` will not be evicted since it was created and admitted on the cluster before the LimitRange creation. - - -## Limiting Storage resources - -You can enforce minimum and maximum size of [storage resources](/docs/concepts/storage/persistent-volumes/) that can be requested by each PersistentVolumeClaim in a namespace using a LimitRange. - -{{< codenew file="admin/resource/storagelimits.yaml" >}} - -Apply the YAML using `kubectl create`. - -```shell -kubectl create -f https://k8s.io/examples/admin/resource/storagelimits.yaml -n limitrange-demo -``` - -```shell -limitrange/storagelimits created -``` -Describe the created object, - -```shell -kubectl describe limits/storagelimits -``` -the output should look like - -```shell -Name: storagelimits -Namespace: limitrange-demo -Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio ----- -------- --- --- --------------- ------------- ----------------------- -PersistentVolumeClaim storage 1Gi 2Gi - - - -``` - -{{< codenew file="admin/resource/pvc-limit-lower.yaml" >}} - -```shell -kubectl create -f https://k8s.io/examples/admin/resource//pvc-limit-lower.yaml -n limitrange-demo -``` - -While creating a PVC with `requests.storage` lower than the Min value in the LimitRange, an Error thrown by the server - -```shell -Error from server (Forbidden): error when creating "pvc-limit-lower.yaml": persistentvolumeclaims "pvc-limit-lower" is forbidden: minimum storage usage per PersistentVolumeClaim is 1Gi, but request is 500Mi. -``` - -Same behaviour is noted if the `requests.storage` is greater than the Max value in the LimitRange - -{{< codenew file="admin/resource/pvc-limit-greater.yaml" >}} - -```shell -kubectl create -f https://k8s.io/examples/admin/resource/pvc-limit-greater.yaml -n limitrange-demo -``` - -```shell -Error from server (Forbidden): error when creating "pvc-limit-greater.yaml": persistentvolumeclaims "pvc-limit-greater" is forbidden: maximum storage usage per PersistentVolumeClaim is 2Gi, but request is 5Gi. -``` - -## Limits/Requests Ratio - -If `LimitRangeItem.maxLimitRequestRatio` if specified in th `LimitRangeSpec`, the named resource must have a request and limit that are both non-zero where limit divided by request is less than or equal to the enumerated value - - the following `LimitRange` enforces memory limit to be at most twice the amount of the memory request for any pod in the namespace. - -{{< codenew file="admin/resource/limit-memory-ratio-pod.yaml" >}} - -```shell -kubectl apply -f https://k8s.io/examples/admin/resource/limit-memory-ratio-pod.yaml -``` - -Describe the LimitRange with the following kubectl command: - -```shell -$ kubectl describe limitrange/limit-memory-ratio-pod -``` - -```shell -Name: limit-memory-ratio-pod -Namespace: limitrange-demo -Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio ----- -------- --- --- --------------- ------------- ----------------------- -Pod memory - - - - 2 -``` - - -Let's create a pod with `requests.memory=100Mi` and `limits.memory=300Mi` -{{< codenew file="admin/resource/limit-range-pod-3.yaml" >}} - - -```shell -kubectl apply -f https://k8s.io/examples/admin/resource/limit-range-pod-3.yaml -``` - -The pod creation failed as the ratio here (`3`) is greater than the enforced limit (`2`) in `limit-memory-ratio-pod` LimitRange - - -```shell -Error from server (Forbidden): error when creating "limit-range-pod-3.yaml": pods "busybox3" is forbidden: memory max limit to request ratio per Pod is 2, but provided ratio is 3.000000. -``` - - -### Clean up -Delete the `limitrange-demo` namespace to free all resources -```shell -kubectl delete ns limitrange-demo -``` - - -## Examples - -- See [a tutorial on how to limit compute resources per namespace](/docs/tasks/administer-cluster/manage-resources/cpu-constraint-namespace/) . -- Check [how to limit storage consumption](/docs/tasks/administer-cluster/limit-storage-consumption/#limitrange-to-limit-requests-for-storage). -- See a [detailed example on quota per namespace](/docs/tasks/administer-cluster/quota-memory-cpu-namespace/). - -{{% /capture %}} - -{{% capture whatsnext %}} - -See [LimitRanger design doc](https://git.k8s.io/community/contributors/design-proposals/resource-management/admission_control_limit_range.md) for more information. - -{{% /capture %}} +--- +reviewers: +- nelvadas +title: Limit Ranges +content_template: templates/concept +weight: 10 +--- + +{{% capture overview %}} + +By default, containers run with unbounded [compute resources](/docs/user-guide/compute-resources) on a Kubernetes cluster. +With Resource quotas, cluster administrators can restrict the resource consumption and creation on a namespace basis. +Within a namespace, a Pod or Container can consume as much CPU and memory as defined by the namespace's resource quota. There is a concern that one Pod or Container could monopolize all of the resources. Limit Range is a policy to constrain resource by Pod or Container in a namespace. + +{{% /capture %}} + + +{{% capture body %}} + +A limit range, defined by a `LimitRange` object, provides constraints that can: + +- Enforce minimum and maximum compute resources usage per Pod or Container in a namespace. +- Enforce minimum and maximum storage request per PersistentVolumeClaim in a namespace. +- Enforce a ratio between request and limit for a resource in a namespace. +- Set default request/limit for compute resources in a namespace and automatically inject them to Containers at runtime. + +## Enabling Limit Range + +Limit Range support is enabled by default for many Kubernetes distributions. It is +enabled when the apiserver `--enable-admission-plugins=` flag has `LimitRanger` admission controller as +one of its arguments. + +A limit range is enforced in a particular namespace when there is a +`LimitRange` object in that namespace. + +### Overview of Limit Range: + +- The administrator creates one `LimitRange` in one namespace. +- Users create resources like Pods, Containers, and PersistentVolumeClaims in the namespace. +- The `LimitRanger` admission controller enforces defaults limits for all Pods and Container that do not set compute resource requirements and tracks usage to ensure it does not exceed resource minimum , maximum and ratio defined in any `LimitRange` present in the namespace. +- If creating or updating a resource (Pod, Container, PersistentVolumeClaim) violates a limit range constraint, the request to the API server will fail with HTTP status code `403 FORBIDDEN` and a message explaining the constraint that would have been violated. +- If limit range is activated in a namespace for compute resources like `cpu` and `memory`, users must specify + requests or limits for those values; otherwise, the system may reject pod creation. +- LimitRange validations occurs only at Pod Admission stage, not on Running pods. + + +Examples of policies that could be created using limit range are: + +- In a 2 node cluster with a capacity of 8 GiB RAM, and 16 cores, constrain Pods in a namespace to request 100m and not exceeds 500m for CPU , request 200Mi and not exceed 600Mi +- Define default CPU limits and request to 150m and Memory default request to 300Mi for containers started with no cpu and memory requests in their spec. + +In the case where the total limits of the namespace is less than the sum of the limits of the Pods/Containers, +there may be contention for resources; The Containers or Pods will not be created. + +Neither contention nor changes to limitrange will affect already created resources. + +## Limiting Container compute resources + +The following section discusses the creation of a LimitRange acting at Container Level. +A Pod with 04 containers is first created; each container within the Pod has a specific `spec.resource` configuration +each containerwithin the pod is handled differently by the LimitRanger admission controller. + + Create a namespace `limitrange-demo` using the following kubectl command + +```shell +kubectl create namespace limitrange-demo +``` + +To avoid passing the target limitrange-demo in your kubectl commands, change your context with the following command + +```shell +kubectl config set-context --current --namespace=limitrange-demo +``` + +Here is the configuration file for a LimitRange object: +{{< codenew file="admin/resource/limit-mem-cpu-container.yaml" >}} + +This object defines minimum and maximum Memory/CPU limits, default cpu/Memory requests and default limits for CPU/Memory resources to be apply to containers. + +Create the `limit-mem-cpu-per-container` LimitRange in the `limitrange-demo` namespace with the following kubectl command. +```shell +kubectl create -f https://k8s.io/examples/admin/resource/limit-mem-cpu-container.yaml -n limitrange-demo +``` + + +```shell + kubectl describe limitrange/limit-mem-cpu-per-container -n limitrange-demo + ``` + + +```shell +Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio +---- -------- --- --- --------------- ------------- ----------------------- +Container cpu 100m 800m 110m 700m - +Container memory 99Mi 1Gi 111Mi 900Mi - +``` + + + +Here is the configuration file for a Pod with 04 containers to demonstrate LimitRange features : +{{< codenew file="admin/resource/limit-range-pod-1.yaml" >}} + +Create the `busybox1` Pod : + +```shell +kubectl apply -f https://k8s.io/examples/admin/resource/limit-range-pod-1.yaml -n limitrange-demo +``` + +### Container spec with valid CPU/Memory requests and limits +View the the `busybox-cnt01` resource configuration + +```shell +kubectl get po/busybox1 -n limitrange-demo -o json | jq ".spec.containers[0].resources" +``` + +```json +{ + "limits": { + "cpu": "500m", + "memory": "200Mi" + }, + "requests": { + "cpu": "100m", + "memory": "100Mi" + } +} +``` + +- The `busybox-cnt01` Container inside `busybox` Pod defined `requests.cpu=100m` and `requests.memory=100Mi`. +- `100m <= 500m <= 800m` , The container cpu limit (500m) falls inside the authorized CPU limit range. +- `99Mi <= 200Mi <= 1Gi` , The container memory limit (200Mi) falls inside the authorized Memory limit range. +- No request/limits ratio validation for CPU/Memory , thus the container is valid and created. + + +### Container spec with a valid CPU/Memory requests but no limits + +View the `busybox-cnt02` resource configuration + +```shell +kubectl get po/busybox1 -n limitrange-demo -o json | jq ".spec.containers[1].resources" +``` + +```json +{ + "limits": { + "cpu": "700m", + "memory": "900Mi" + }, + "requests": { + "cpu": "100m", + "memory": "100Mi" + } +} +``` +- The `busybox-cnt02` Container inside `busybox1` Pod defined `requests.cpu=100m` and `requests.memory=100Mi` but not limits for cpu and memory. +- The container do not have a limits section, the default limits defined in the limit-mem-cpu-per-container LimitRange object are injected to this container `limits.cpu=700mi` and `limits.memory=900Mi`. +- `100m <= 700m <= 800m` , The container cpu limit (700m) falls inside the authorized CPU limit range. +- `99Mi <= 900Mi <= 1Gi` , The container memory limit (900Mi) falls inside the authorized Memory limit range. +- No request/limits ratio set , thus the container is valid and created. + + +### Container spec with a valid CPU/Memory limits but no requests +View the `busybox-cnt03` resource configuration + +```shell +kubectl get po/busybox1 -n limitrange-demo -o json | jq ".spec.containers[2].resources" +``` +```json +{ + "limits": { + "cpu": "500m", + "memory": "200Mi" + }, + "requests": { + "cpu": "500m", + "memory": "200Mi" + } +} +``` + +- The `busybox-cnt03` Container inside `busybox1` Pod defined `limits.cpu=500m` and `limits.memory=200Mi` but no `requests` for cpu and memory. +- The container do not define a request section, the defaultRequest defined in the limit-mem-cpu-per-container LimitRange is not used to fill its limits section but the limits defined by the container are set as requests `limits.cpu=500m` and `limits.memory=200Mi`. +- `100m <= 500m <= 800m` , The container cpu limit (500m) falls inside the authorized CPU limit range. +- `99Mi <= 200Mi <= 1Gi` , The container memory limit (200Mi) falls inside the authorized Memory limit range. +- No request/limits ratio set , thus the container is valid and created. + + + +### Container spec with no CPU/Memory requests/limits +View the `busybox-cnt04` resource configuration +```shell +kubectl get po/busybox1 -n limitrange-demo -o json | jq ".spec.containers[3].resources" +``` +```json +{ + "limits": { + "cpu": "700m", + "memory": "900Mi" + }, + "requests": { + "cpu": "110m", + "memory": "111Mi" + } +} +``` + +- The `busybox-cnt04` Container inside `busybox1` define neither `limits` nor `requests`. +- The container do not define a limit section, the default limit defined in the limit-mem-cpu-per-container LimitRange is used to fill its request + `limits.cpu=700m and` `limits.memory=900Mi` . +- The container do not define a request section, the defaultRequest defined in the limit-mem-cpu-per-container LimitRange is used to fill its request section requests.cpu=110m and requests.memory=111Mi +- `100m <= 700m <= 800m` , The container cpu limit (700m) falls inside the authorized CPU limit range. +- `99Mi <= 900Mi <= 1Gi` , The container memory limit (900Mi) falls inside the authorized Memory limitrange . +- No request/limits ratio set , thus the container is valid and created. + +All containers defined in the `busybox` Pod passed LimitRange validations, this the Pod is valid and create in the namespace. + +## Limiting Pod compute resources +The following section discusses how to constrain resources at Pod level. + +{{< codenew file="admin/resource/limit-mem-cpu-pod.yaml" >}} + +Without having to delete `busybox1` Pod, create the `limit-mem-cpu-pod` LimitRange in the `limitrange-demo` namespace +```shell +kubectl apply -f https://k8s.io/examples/admin/resource/limit-mem-cpu-pod.yaml -n limitrange-demo +``` +The limitrange is created and limits CPU to 2 Core and Memory to 2Gi per Pod. +```shell +limitrange/limit-mem-cpu-per-pod created +``` +Describe the `limit-mem-cpu-per-pod` limit object using the following kubectl command +```shell +kubectl describe limitrange/limit-mem-cpu-per-pod +``` + +```shell +Name: limit-mem-cpu-per-pod +Namespace: limitrange-demo +Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio +---- -------- --- --- --------------- ------------- ----------------------- +Pod cpu - 2 - - - +Pod memory - 2Gi - - - +``` +Now create the `busybox2` Pod. + +{{< codenew file="admin/resource/limit-range-pod-2.yaml" >}} + +```shell +kubectl apply -f https://k8s.io/examples/admin/resource/limit-range-pod-2.yaml -n limitrange-demo +``` +The `busybox2` Pod definition is identical to `busybox1` but an error is reported since Pod's resources are now limited +```shell +Error from server (Forbidden): error when creating "limit-range-pod-2.yaml": pods "busybox2" is forbidden: [maximum cpu usage per Pod is 2, but limit is 2400m., maximum memory usage per Pod is 2Gi, but limit is 2306867200.] +``` + +```shell +kubectl get po/busybox1 -n limitrange-demo -o json | jq ".spec.containers[].resources.limits.memory" +"200Mi" +"900Mi" +"200Mi" +"900Mi" +``` +`busybox2` Pod will not be admitted on the cluster since the total memory limit of its container is greater than the limit defined in the LimitRange. +`busybox1` will not be evicted since it was created and admitted on the cluster before the LimitRange creation. + + +## Limiting Storage resources + +You can enforce minimum and maximum size of [storage resources](/docs/concepts/storage/persistent-volumes/) that can be requested by each PersistentVolumeClaim in a namespace using a LimitRange. + +{{< codenew file="admin/resource/storagelimits.yaml" >}} + +Apply the YAML using `kubectl create`. + +```shell +kubectl create -f https://k8s.io/examples/admin/resource/storagelimits.yaml -n limitrange-demo +``` + +```shell +limitrange/storagelimits created +``` +Describe the created object, + +```shell +kubectl describe limits/storagelimits +``` +the output should look like + +```shell +Name: storagelimits +Namespace: limitrange-demo +Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio +---- -------- --- --- --------------- ------------- ----------------------- +PersistentVolumeClaim storage 1Gi 2Gi - - - +``` + +{{< codenew file="admin/resource/pvc-limit-lower.yaml" >}} + +```shell +kubectl create -f https://k8s.io/examples/admin/resource//pvc-limit-lower.yaml -n limitrange-demo +``` + +While creating a PVC with `requests.storage` lower than the Min value in the LimitRange, an Error thrown by the server + +```shell +Error from server (Forbidden): error when creating "pvc-limit-lower.yaml": persistentvolumeclaims "pvc-limit-lower" is forbidden: minimum storage usage per PersistentVolumeClaim is 1Gi, but request is 500Mi. +``` + +Same behaviour is noted if the `requests.storage` is greater than the Max value in the LimitRange + +{{< codenew file="admin/resource/pvc-limit-greater.yaml" >}} + +```shell +kubectl create -f https://k8s.io/examples/admin/resource/pvc-limit-greater.yaml -n limitrange-demo +``` + +```shell +Error from server (Forbidden): error when creating "pvc-limit-greater.yaml": persistentvolumeclaims "pvc-limit-greater" is forbidden: maximum storage usage per PersistentVolumeClaim is 2Gi, but request is 5Gi. +``` + +## Limits/Requests Ratio + +If `LimitRangeItem.maxLimitRequestRatio` if specified in th `LimitRangeSpec`, the named resource must have a request and limit that are both non-zero where limit divided by request is less than or equal to the enumerated value + + the following `LimitRange` enforces memory limit to be at most twice the amount of the memory request for any pod in the namespace. + +{{< codenew file="admin/resource/limit-memory-ratio-pod.yaml" >}} + +```shell +kubectl apply -f https://k8s.io/examples/admin/resource/limit-memory-ratio-pod.yaml +``` + +Describe the LimitRange with the following kubectl command: + +```shell +$ kubectl describe limitrange/limit-memory-ratio-pod +``` + +```shell +Name: limit-memory-ratio-pod +Namespace: limitrange-demo +Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio +---- -------- --- --- --------------- ------------- ----------------------- +Pod memory - - - - 2 +``` + + +Let's create a pod with `requests.memory=100Mi` and `limits.memory=300Mi` +{{< codenew file="admin/resource/limit-range-pod-3.yaml" >}} + + +```shell +kubectl apply -f https://k8s.io/examples/admin/resource/limit-range-pod-3.yaml +``` + +The pod creation failed as the ratio here (`3`) is greater than the enforced limit (`2`) in `limit-memory-ratio-pod` LimitRange + + +```shell +Error from server (Forbidden): error when creating "limit-range-pod-3.yaml": pods "busybox3" is forbidden: memory max limit to request ratio per Pod is 2, but provided ratio is 3.000000. +``` + + +### Clean up +Delete the `limitrange-demo` namespace to free all resources +```shell +kubectl delete ns limitrange-demo +``` + + +## Examples + +- See [a tutorial on how to limit compute resources per namespace](/docs/tasks/administer-cluster/manage-resources/cpu-constraint-namespace/) . +- Check [how to limit storage consumption](/docs/tasks/administer-cluster/limit-storage-consumption/#limitrange-to-limit-requests-for-storage). +- See a [detailed example on quota per namespace](/docs/tasks/administer-cluster/quota-memory-cpu-namespace/). + +{{% /capture %}} + +{{% capture whatsnext %}} + +See [LimitRanger design doc](https://git.k8s.io/community/contributors/design-proposals/resource-management/admission_control_limit_range.md) for more information. + +{{% /capture %}} diff --git a/content/en/docs/concepts/policy/resource-quotas.md b/content/en/docs/concepts/policy/resource-quotas.md index 3150d5cf2a..c4f1f4a240 100644 --- a/content/en/docs/concepts/policy/resource-quotas.md +++ b/content/en/docs/concepts/policy/resource-quotas.md @@ -1,578 +1,578 @@ ---- -reviewers: -- derekwaynecarr -title: Resource Quotas -content_template: templates/concept -weight: 10 ---- - -{{% capture overview %}} - -When several users or teams share a cluster with a fixed number of nodes, -there is a concern that one team could use more than its fair share of resources. - -Resource quotas are a tool for administrators to address this concern. - -{{% /capture %}} - - -{{% capture body %}} - -A resource quota, defined by a `ResourceQuota` object, provides constraints that limit -aggregate resource consumption per namespace. It can limit the quantity of objects that can -be created in a namespace by type, as well as the total amount of compute resources that may -be consumed by resources in that project. - -Resource quotas work like this: - -- Different teams work in different namespaces. Currently this is voluntary, but - support for making this mandatory via ACLs is planned. -- The administrator creates one `ResourceQuota` for each namespace. -- Users create resources (pods, services, etc.) in the namespace, and the quota system - tracks usage to ensure it does not exceed hard resource limits defined in a `ResourceQuota`. -- If creating or updating a resource violates a quota constraint, the request will fail with HTTP - status code `403 FORBIDDEN` with a message explaining the constraint that would have been violated. -- If quota is enabled in a namespace for compute resources like `cpu` and `memory`, users must specify - requests or limits for those values; otherwise, the quota system may reject pod creation. Hint: Use - the `LimitRanger` admission controller to force defaults for pods that make no compute resource requirements. - See the [walkthrough](/docs/tasks/administer-cluster/quota-memory-cpu-namespace/) for an example of how to avoid this problem. - -Examples of policies that could be created using namespaces and quotas are: - -- In a cluster with a capacity of 32 GiB RAM, and 16 cores, let team A use 20 GiB and 10 cores, - let B use 10GiB and 4 cores, and hold 2GiB and 2 cores in reserve for future allocation. -- Limit the "testing" namespace to using 1 core and 1GiB RAM. Let the "production" namespace - use any amount. - -In the case where the total capacity of the cluster is less than the sum of the quotas of the namespaces, -there may be contention for resources. This is handled on a first-come-first-served basis. - -Neither contention nor changes to quota will affect already created resources. - -## Enabling Resource Quota - -Resource Quota support is enabled by default for many Kubernetes distributions. It is -enabled when the apiserver `--enable-admission-plugins=` flag has `ResourceQuota` as -one of its arguments. - -A resource quota is enforced in a particular namespace when there is a -`ResourceQuota` in that namespace. - -## Compute Resource Quota - -You can limit the total sum of [compute resources](/docs/user-guide/compute-resources) that can be requested in a given namespace. - -The following resource types are supported: - -| Resource Name | Description | -| --------------------- | ----------------------------------------------------------- | -| `limits.cpu` | Across all pods in a non-terminal state, the sum of CPU limits cannot exceed this value. | -| `limits.memory` | Across all pods in a non-terminal state, the sum of memory limits cannot exceed this value. | -| `requests.cpu` | Across all pods in a non-terminal state, the sum of CPU requests cannot exceed this value. | -| `requests.memory` | Across all pods in a non-terminal state, the sum of memory requests cannot exceed this value. | - -### Resource Quota For Extended Resources - -In addition to the resources mentioned above, in release 1.10, quota support for -[extended resources](/docs/concepts/configuration/manage-compute-resources-container/#extended-resources) is added. - -As overcommit is not allowed for extended resources, it makes no sense to specify both `requests` -and `limits` for the same extended resource in a quota. So for extended resources, only quota items -with prefix `requests.` is allowed for now. - -Take the GPU resource as an example, if the resource name is `nvidia.com/gpu`, and you want to -limit the total number of GPUs requested in a namespace to 4, you can define a quota as follows: - -* `requests.nvidia.com/gpu: 4` - -See [Viewing and Setting Quotas](#viewing-and-setting-quotas) for more detail information. - - -## Storage Resource Quota - -You can limit the total sum of [storage resources](/docs/concepts/storage/persistent-volumes/) that can be requested in a given namespace. - -In addition, you can limit consumption of storage resources based on associated storage-class. - -| Resource Name | Description | -| --------------------- | ----------------------------------------------------------- | -| `requests.storage` | Across all persistent volume claims, the sum of storage requests cannot exceed this value. | -| `persistentvolumeclaims` | The total number of [persistent volume claims](/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims) that can exist in the namespace. | -| `.storageclass.storage.k8s.io/requests.storage` | Across all persistent volume claims associated with the storage-class-name, the sum of storage requests cannot exceed this value. | -| `.storageclass.storage.k8s.io/persistentvolumeclaims` | Across all persistent volume claims associated with the storage-class-name, the total number of [persistent volume claims](/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims) that can exist in the namespace. | - -For example, if an operator wants to quota storage with `gold` storage class separate from `bronze` storage class, the operator can -define a quota as follows: - -* `gold.storageclass.storage.k8s.io/requests.storage: 500Gi` -* `bronze.storageclass.storage.k8s.io/requests.storage: 100Gi` - -In release 1.8, quota support for local ephemeral storage is added as an alpha feature: - -| Resource Name | Description | -| ------------------------------- |----------------------------------------------------------- | -| `requests.ephemeral-storage` | Across all pods in the namespace, the sum of local ephemeral storage requests cannot exceed this value. | -| `limits.ephemeral-storage` | Across all pods in the namespace, the sum of local ephemeral storage limits cannot exceed this value. | - -## Object Count Quota - -The 1.9 release added support to quota all standard namespaced resource types using the following syntax: - -* `count/.` - -Here is an example set of resources users may want to put under object count quota: - -* `count/persistentvolumeclaims` -* `count/services` -* `count/secrets` -* `count/configmaps` -* `count/replicationcontrollers` -* `count/deployments.apps` -* `count/replicasets.apps` -* `count/statefulsets.apps` -* `count/jobs.batch` -* `count/cronjobs.batch` -* `count/deployments.extensions` - -The 1.15 release added support for custom resources using the same syntax. -For example, to create a quota on a `widgets` custom resource in the `example.com` API group, use `count/widgets.example.com`. - -When using `count/*` resource quota, an object is charged against the quota if it exists in server storage. -These types of quotas are useful to protect against exhaustion of storage resources. For example, you may -want to quota the number of secrets in a server given their large size. Too many secrets in a cluster can -actually prevent servers and controllers from starting! You may choose to quota jobs to protect against -a poorly configured cronjob creating too many jobs in a namespace causing a denial of service. - -Prior to the 1.9 release, it was possible to do generic object count quota on a limited set of resources. -In addition, it is possible to further constrain quota for particular resources by their type. - -The following types are supported: - -| Resource Name | Description | -| ------------------------------- | ------------------------------------------------- | -| `configmaps` | The total number of config maps that can exist in the namespace. | -| `persistentvolumeclaims` | The total number of [persistent volume claims](/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims) that can exist in the namespace. | -| `pods` | The total number of pods in a non-terminal state that can exist in the namespace. A pod is in a terminal state if `.status.phase in (Failed, Succeeded)` is true. | -| `replicationcontrollers` | The total number of replication controllers that can exist in the namespace. | -| `resourcequotas` | The total number of [resource quotas](/docs/reference/access-authn-authz/admission-controllers/#resourcequota) that can exist in the namespace. | -| `services` | The total number of services that can exist in the namespace. | -| `services.loadbalancers` | The total number of services of type load balancer that can exist in the namespace. | -| `services.nodeports` | The total number of services of type node port that can exist in the namespace. | -| `secrets` | The total number of secrets that can exist in the namespace. | - -For example, `pods` quota counts and enforces a maximum on the number of `pods` -created in a single namespace that are not terminal. You might want to set a `pods` -quota on a namespace to avoid the case where a user creates many small pods and -exhausts the cluster's supply of Pod IPs. - -## Quota Scopes - -Each quota can have an associated set of scopes. A quota will only measure usage for a resource if it matches -the intersection of enumerated scopes. - -When a scope is added to the quota, it limits the number of resources it supports to those that pertain to the scope. -Resources specified on the quota outside of the allowed set results in a validation error. - -| Scope | Description | -| ----- | ----------- | -| `Terminating` | Match pods where `.spec.activeDeadlineSeconds >= 0` | -| `NotTerminating` | Match pods where `.spec.activeDeadlineSeconds is nil` | -| `BestEffort` | Match pods that have best effort quality of service. | -| `NotBestEffort` | Match pods that do not have best effort quality of service. | - -The `BestEffort` scope restricts a quota to tracking the following resource: `pods` - -The `Terminating`, `NotTerminating`, and `NotBestEffort` scopes restrict a quota to tracking the following resources: - -* `cpu` -* `limits.cpu` -* `limits.memory` -* `memory` -* `pods` -* `requests.cpu` -* `requests.memory` - -### Resource Quota Per PriorityClass - -{{< feature-state for_k8s_version="1.12" state="beta" >}} - -Pods can be created at a specific [priority](/docs/concepts/configuration/pod-priority-preemption/#pod-priority). -You can control a pod's consumption of system resources based on a pod's priority, by using the `scopeSelector` -field in the quota spec. - -A quota is matched and consumed only if `scopeSelector` in the quota spec selects the pod. - -This example creates a quota object and matches it with pods at specific priorities. The example -works as follows: - -- Pods in the cluster have one of the three priority classes, "low", "medium", "high". -- One quota object is created for each priority. - -Save the following YAML to a file `quota.yml`. - -```yaml -apiVersion: v1 -kind: List -items: -- apiVersion: v1 - kind: ResourceQuota - metadata: - name: pods-high - spec: - hard: - cpu: "1000" - memory: 200Gi - pods: "10" - scopeSelector: - matchExpressions: - - operator : In - scopeName: PriorityClass - values: ["high"] -- apiVersion: v1 - kind: ResourceQuota - metadata: - name: pods-medium - spec: - hard: - cpu: "10" - memory: 20Gi - pods: "10" - scopeSelector: - matchExpressions: - - operator : In - scopeName: PriorityClass - values: ["medium"] -- apiVersion: v1 - kind: ResourceQuota - metadata: - name: pods-low - spec: - hard: - cpu: "5" - memory: 10Gi - pods: "10" - scopeSelector: - matchExpressions: - - operator : In - scopeName: PriorityClass - values: ["low"] -``` - -Apply the YAML using `kubectl create`. - -```shell -kubectl create -f ./quota.yml -``` - -```shell -resourcequota/pods-high created -resourcequota/pods-medium created -resourcequota/pods-low created -``` - -Verify that `Used` quota is `0` using `kubectl describe quota`. - -```shell -kubectl describe quota -``` - -```shell -Name: pods-high -Namespace: default -Resource Used Hard --------- ---- ---- -cpu 0 1k -memory 0 200Gi -pods 0 10 - - -Name: pods-low -Namespace: default -Resource Used Hard --------- ---- ---- -cpu 0 5 -memory 0 10Gi -pods 0 10 - - -Name: pods-medium -Namespace: default -Resource Used Hard --------- ---- ---- -cpu 0 10 -memory 0 20Gi -pods 0 10 -``` - -Create a pod with priority "high". Save the following YAML to a -file `high-priority-pod.yml`. - -```yaml -apiVersion: v1 -kind: Pod -metadata: - name: high-priority -spec: - containers: - - name: high-priority - image: ubuntu - command: ["/bin/sh"] - args: ["-c", "while true; do echo hello; sleep 10;done"] - resources: - requests: - memory: "10Gi" - cpu: "500m" - limits: - memory: "10Gi" - cpu: "500m" - priorityClassName: high -``` - -Apply it with `kubectl create`. - -```shell -kubectl create -f ./high-priority-pod.yml -``` - -Verify that "Used" stats for "high" priority quota, `pods-high`, has changed and that -the other two quotas are unchanged. - -```shell -kubectl describe quota -``` - -```shell -Name: pods-high -Namespace: default -Resource Used Hard --------- ---- ---- -cpu 500m 1k -memory 10Gi 200Gi -pods 1 10 - - -Name: pods-low -Namespace: default -Resource Used Hard --------- ---- ---- -cpu 0 5 -memory 0 10Gi -pods 0 10 - - -Name: pods-medium -Namespace: default -Resource Used Hard --------- ---- ---- -cpu 0 10 -memory 0 20Gi -pods 0 10 -``` - -`scopeSelector` supports the following values in the `operator` field: - -* `In` -* `NotIn` -* `Exist` -* `DoesNotExist` - -## Requests vs Limits - -When allocating compute resources, each container may specify a request and a limit value for either CPU or memory. -The quota can be configured to quota either value. - -If the quota has a value specified for `requests.cpu` or `requests.memory`, then it requires that every incoming -container makes an explicit request for those resources. If the quota has a value specified for `limits.cpu` or `limits.memory`, -then it requires that every incoming container specifies an explicit limit for those resources. - -## Viewing and Setting Quotas - -Kubectl supports creating, updating, and viewing quotas: - -```shell -kubectl create namespace myspace -``` - -```shell -cat < compute-resources.yaml -apiVersion: v1 -kind: ResourceQuota -metadata: - name: compute-resources -spec: - hard: - pods: "4" - requests.cpu: "1" - requests.memory: 1Gi - limits.cpu: "2" - limits.memory: 2Gi - requests.nvidia.com/gpu: 4 -EOF -``` - -```shell -kubectl create -f ./compute-resources.yaml --namespace=myspace -``` - -```shell -cat < object-counts.yaml -apiVersion: v1 -kind: ResourceQuota -metadata: - name: object-counts -spec: - hard: - configmaps: "10" - persistentvolumeclaims: "4" - replicationcontrollers: "20" - secrets: "10" - services: "10" - services.loadbalancers: "2" -EOF -``` - -```shell -kubectl create -f ./object-counts.yaml --namespace=myspace -``` - -```shell -kubectl get quota --namespace=myspace -``` - -```shell -NAME AGE -compute-resources 30s -object-counts 32s -``` - -```shell -kubectl describe quota compute-resources --namespace=myspace -``` - -```shell -Name: compute-resources -Namespace: myspace -Resource Used Hard --------- ---- ---- -limits.cpu 0 2 -limits.memory 0 2Gi -pods 0 4 -requests.cpu 0 1 -requests.memory 0 1Gi -requests.nvidia.com/gpu 0 4 -``` - -```shell -kubectl describe quota object-counts --namespace=myspace -``` - -```shell -Name: object-counts -Namespace: myspace -Resource Used Hard --------- ---- ---- -configmaps 0 10 -persistentvolumeclaims 0 4 -replicationcontrollers 0 20 -secrets 1 10 -services 0 10 -services.loadbalancers 0 2 -``` - -Kubectl also supports object count quota for all standard namespaced resources -using the syntax `count/.`: - -```shell -kubectl create namespace myspace -``` - -```shell -kubectl create quota test --hard=count/deployments.extensions=2,count/replicasets.extensions=4,count/pods=3,count/secrets=4 --namespace=myspace -``` - -```shell -kubectl run nginx --image=nginx --replicas=2 --namespace=myspace -``` - -```shell -kubectl describe quota --namespace=myspace -``` - -```shell -Name: test -Namespace: myspace -Resource Used Hard --------- ---- ---- -count/deployments.extensions 1 2 -count/pods 2 3 -count/replicasets.extensions 1 4 -count/secrets 1 4 -``` - -## Quota and Cluster Capacity - -`ResourceQuotas` are independent of the cluster capacity. They are -expressed in absolute units. So, if you add nodes to your cluster, this does *not* -automatically give each namespace the ability to consume more resources. - -Sometimes more complex policies may be desired, such as: - - - Proportionally divide total cluster resources among several teams. - - Allow each tenant to grow resource usage as needed, but have a generous - limit to prevent accidental resource exhaustion. - - Detect demand from one namespace, add nodes, and increase quota. - -Such policies could be implemented using `ResourceQuotas` as building blocks, by -writing a "controller" that watches the quota usage and adjusts the quota -hard limits of each namespace according to other signals. - -Note that resource quota divides up aggregate cluster resources, but it creates no -restrictions around nodes: pods from several namespaces may run on the same node. - -## Limit Priority Class consumption by default - -It may be desired that pods at a particular priority, eg. "cluster-services", should be allowed in a namespace, if and only if, a matching quota object exists. - -With this mechanism, operators will be able to restrict usage of certain high priority classes to a limited number of namespaces and not every namespace will be able to consume these priority classes by default. - -To enforce this, kube-apiserver flag `--admission-control-config-file` should be used to pass path to the following configuration file: - -```yaml -apiVersion: apiserver.k8s.io/v1alpha1 -kind: AdmissionConfiguration -plugins: -- name: "ResourceQuota" - configuration: - apiVersion: resourcequota.admission.k8s.io/v1beta1 - kind: Configuration - limitedResources: - - resource: pods - matchScopes: - - scopeName: PriorityClass - operator: In - values: ["cluster-services"] -``` - -Now, "cluster-services" pods will be allowed in only those namespaces where a quota object with a matching `scopeSelector` is present. -For example: -```yaml - scopeSelector: - matchExpressions: - - scopeName: PriorityClass - operator: In - values: ["cluster-services"] -``` - -See [LimitedResources](https://github.com/kubernetes/kubernetes/pull/36765) and [Quota support for priority class design doc](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/scheduling/pod-priority-resourcequota.md) for more information. - -## Example - -See a [detailed example for how to use resource quota](/docs/tasks/administer-cluster/quota-api-object/). - -{{% /capture %}} - -{{% capture whatsnext %}} - -See [ResourceQuota design doc](https://git.k8s.io/community/contributors/design-proposals/resource-management/admission_control_resource_quota.md) for more information. - -{{% /capture %}} +--- +reviewers: +- derekwaynecarr +title: Resource Quotas +content_template: templates/concept +weight: 10 +--- + +{{% capture overview %}} + +When several users or teams share a cluster with a fixed number of nodes, +there is a concern that one team could use more than its fair share of resources. + +Resource quotas are a tool for administrators to address this concern. + +{{% /capture %}} + + +{{% capture body %}} + +A resource quota, defined by a `ResourceQuota` object, provides constraints that limit +aggregate resource consumption per namespace. It can limit the quantity of objects that can +be created in a namespace by type, as well as the total amount of compute resources that may +be consumed by resources in that project. + +Resource quotas work like this: + +- Different teams work in different namespaces. Currently this is voluntary, but + support for making this mandatory via ACLs is planned. +- The administrator creates one `ResourceQuota` for each namespace. +- Users create resources (pods, services, etc.) in the namespace, and the quota system + tracks usage to ensure it does not exceed hard resource limits defined in a `ResourceQuota`. +- If creating or updating a resource violates a quota constraint, the request will fail with HTTP + status code `403 FORBIDDEN` with a message explaining the constraint that would have been violated. +- If quota is enabled in a namespace for compute resources like `cpu` and `memory`, users must specify + requests or limits for those values; otherwise, the quota system may reject pod creation. Hint: Use + the `LimitRanger` admission controller to force defaults for pods that make no compute resource requirements. + See the [walkthrough](/docs/tasks/administer-cluster/quota-memory-cpu-namespace/) for an example of how to avoid this problem. + +Examples of policies that could be created using namespaces and quotas are: + +- In a cluster with a capacity of 32 GiB RAM, and 16 cores, let team A use 20 GiB and 10 cores, + let B use 10GiB and 4 cores, and hold 2GiB and 2 cores in reserve for future allocation. +- Limit the "testing" namespace to using 1 core and 1GiB RAM. Let the "production" namespace + use any amount. + +In the case where the total capacity of the cluster is less than the sum of the quotas of the namespaces, +there may be contention for resources. This is handled on a first-come-first-served basis. + +Neither contention nor changes to quota will affect already created resources. + +## Enabling Resource Quota + +Resource Quota support is enabled by default for many Kubernetes distributions. It is +enabled when the apiserver `--enable-admission-plugins=` flag has `ResourceQuota` as +one of its arguments. + +A resource quota is enforced in a particular namespace when there is a +`ResourceQuota` in that namespace. + +## Compute Resource Quota + +You can limit the total sum of [compute resources](/docs/user-guide/compute-resources) that can be requested in a given namespace. + +The following resource types are supported: + +| Resource Name | Description | +| --------------------- | ----------------------------------------------------------- | +| `limits.cpu` | Across all pods in a non-terminal state, the sum of CPU limits cannot exceed this value. | +| `limits.memory` | Across all pods in a non-terminal state, the sum of memory limits cannot exceed this value. | +| `requests.cpu` | Across all pods in a non-terminal state, the sum of CPU requests cannot exceed this value. | +| `requests.memory` | Across all pods in a non-terminal state, the sum of memory requests cannot exceed this value. | + +### Resource Quota For Extended Resources + +In addition to the resources mentioned above, in release 1.10, quota support for +[extended resources](/docs/concepts/configuration/manage-compute-resources-container/#extended-resources) is added. + +As overcommit is not allowed for extended resources, it makes no sense to specify both `requests` +and `limits` for the same extended resource in a quota. So for extended resources, only quota items +with prefix `requests.` is allowed for now. + +Take the GPU resource as an example, if the resource name is `nvidia.com/gpu`, and you want to +limit the total number of GPUs requested in a namespace to 4, you can define a quota as follows: + +* `requests.nvidia.com/gpu: 4` + +See [Viewing and Setting Quotas](#viewing-and-setting-quotas) for more detail information. + + +## Storage Resource Quota + +You can limit the total sum of [storage resources](/docs/concepts/storage/persistent-volumes/) that can be requested in a given namespace. + +In addition, you can limit consumption of storage resources based on associated storage-class. + +| Resource Name | Description | +| --------------------- | ----------------------------------------------------------- | +| `requests.storage` | Across all persistent volume claims, the sum of storage requests cannot exceed this value. | +| `persistentvolumeclaims` | The total number of [persistent volume claims](/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims) that can exist in the namespace. | +| `.storageclass.storage.k8s.io/requests.storage` | Across all persistent volume claims associated with the storage-class-name, the sum of storage requests cannot exceed this value. | +| `.storageclass.storage.k8s.io/persistentvolumeclaims` | Across all persistent volume claims associated with the storage-class-name, the total number of [persistent volume claims](/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims) that can exist in the namespace. | + +For example, if an operator wants to quota storage with `gold` storage class separate from `bronze` storage class, the operator can +define a quota as follows: + +* `gold.storageclass.storage.k8s.io/requests.storage: 500Gi` +* `bronze.storageclass.storage.k8s.io/requests.storage: 100Gi` + +In release 1.8, quota support for local ephemeral storage is added as an alpha feature: + +| Resource Name | Description | +| ------------------------------- |----------------------------------------------------------- | +| `requests.ephemeral-storage` | Across all pods in the namespace, the sum of local ephemeral storage requests cannot exceed this value. | +| `limits.ephemeral-storage` | Across all pods in the namespace, the sum of local ephemeral storage limits cannot exceed this value. | + +## Object Count Quota + +The 1.9 release added support to quota all standard namespaced resource types using the following syntax: + +* `count/.` + +Here is an example set of resources users may want to put under object count quota: + +* `count/persistentvolumeclaims` +* `count/services` +* `count/secrets` +* `count/configmaps` +* `count/replicationcontrollers` +* `count/deployments.apps` +* `count/replicasets.apps` +* `count/statefulsets.apps` +* `count/jobs.batch` +* `count/cronjobs.batch` +* `count/deployments.extensions` + +The 1.15 release added support for custom resources using the same syntax. +For example, to create a quota on a `widgets` custom resource in the `example.com` API group, use `count/widgets.example.com`. + +When using `count/*` resource quota, an object is charged against the quota if it exists in server storage. +These types of quotas are useful to protect against exhaustion of storage resources. For example, you may +want to quota the number of secrets in a server given their large size. Too many secrets in a cluster can +actually prevent servers and controllers from starting! You may choose to quota jobs to protect against +a poorly configured cronjob creating too many jobs in a namespace causing a denial of service. + +Prior to the 1.9 release, it was possible to do generic object count quota on a limited set of resources. +In addition, it is possible to further constrain quota for particular resources by their type. + +The following types are supported: + +| Resource Name | Description | +| ------------------------------- | ------------------------------------------------- | +| `configmaps` | The total number of config maps that can exist in the namespace. | +| `persistentvolumeclaims` | The total number of [persistent volume claims](/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims) that can exist in the namespace. | +| `pods` | The total number of pods in a non-terminal state that can exist in the namespace. A pod is in a terminal state if `.status.phase in (Failed, Succeeded)` is true. | +| `replicationcontrollers` | The total number of replication controllers that can exist in the namespace. | +| `resourcequotas` | The total number of [resource quotas](/docs/reference/access-authn-authz/admission-controllers/#resourcequota) that can exist in the namespace. | +| `services` | The total number of services that can exist in the namespace. | +| `services.loadbalancers` | The total number of services of type load balancer that can exist in the namespace. | +| `services.nodeports` | The total number of services of type node port that can exist in the namespace. | +| `secrets` | The total number of secrets that can exist in the namespace. | + +For example, `pods` quota counts and enforces a maximum on the number of `pods` +created in a single namespace that are not terminal. You might want to set a `pods` +quota on a namespace to avoid the case where a user creates many small pods and +exhausts the cluster's supply of Pod IPs. + +## Quota Scopes + +Each quota can have an associated set of scopes. A quota will only measure usage for a resource if it matches +the intersection of enumerated scopes. + +When a scope is added to the quota, it limits the number of resources it supports to those that pertain to the scope. +Resources specified on the quota outside of the allowed set results in a validation error. + +| Scope | Description | +| ----- | ----------- | +| `Terminating` | Match pods where `.spec.activeDeadlineSeconds >= 0` | +| `NotTerminating` | Match pods where `.spec.activeDeadlineSeconds is nil` | +| `BestEffort` | Match pods that have best effort quality of service. | +| `NotBestEffort` | Match pods that do not have best effort quality of service. | + +The `BestEffort` scope restricts a quota to tracking the following resource: `pods` + +The `Terminating`, `NotTerminating`, and `NotBestEffort` scopes restrict a quota to tracking the following resources: + +* `cpu` +* `limits.cpu` +* `limits.memory` +* `memory` +* `pods` +* `requests.cpu` +* `requests.memory` + +### Resource Quota Per PriorityClass + +{{< feature-state for_k8s_version="1.12" state="beta" >}} + +Pods can be created at a specific [priority](/docs/concepts/configuration/pod-priority-preemption/#pod-priority). +You can control a pod's consumption of system resources based on a pod's priority, by using the `scopeSelector` +field in the quota spec. + +A quota is matched and consumed only if `scopeSelector` in the quota spec selects the pod. + +This example creates a quota object and matches it with pods at specific priorities. The example +works as follows: + +- Pods in the cluster have one of the three priority classes, "low", "medium", "high". +- One quota object is created for each priority. + +Save the following YAML to a file `quota.yml`. + +```yaml +apiVersion: v1 +kind: List +items: +- apiVersion: v1 + kind: ResourceQuota + metadata: + name: pods-high + spec: + hard: + cpu: "1000" + memory: 200Gi + pods: "10" + scopeSelector: + matchExpressions: + - operator : In + scopeName: PriorityClass + values: ["high"] +- apiVersion: v1 + kind: ResourceQuota + metadata: + name: pods-medium + spec: + hard: + cpu: "10" + memory: 20Gi + pods: "10" + scopeSelector: + matchExpressions: + - operator : In + scopeName: PriorityClass + values: ["medium"] +- apiVersion: v1 + kind: ResourceQuota + metadata: + name: pods-low + spec: + hard: + cpu: "5" + memory: 10Gi + pods: "10" + scopeSelector: + matchExpressions: + - operator : In + scopeName: PriorityClass + values: ["low"] +``` + +Apply the YAML using `kubectl create`. + +```shell +kubectl create -f ./quota.yml +``` + +```shell +resourcequota/pods-high created +resourcequota/pods-medium created +resourcequota/pods-low created +``` + +Verify that `Used` quota is `0` using `kubectl describe quota`. + +```shell +kubectl describe quota +``` + +```shell +Name: pods-high +Namespace: default +Resource Used Hard +-------- ---- ---- +cpu 0 1k +memory 0 200Gi +pods 0 10 + + +Name: pods-low +Namespace: default +Resource Used Hard +-------- ---- ---- +cpu 0 5 +memory 0 10Gi +pods 0 10 + + +Name: pods-medium +Namespace: default +Resource Used Hard +-------- ---- ---- +cpu 0 10 +memory 0 20Gi +pods 0 10 +``` + +Create a pod with priority "high". Save the following YAML to a +file `high-priority-pod.yml`. + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: high-priority +spec: + containers: + - name: high-priority + image: ubuntu + command: ["/bin/sh"] + args: ["-c", "while true; do echo hello; sleep 10;done"] + resources: + requests: + memory: "10Gi" + cpu: "500m" + limits: + memory: "10Gi" + cpu: "500m" + priorityClassName: high +``` + +Apply it with `kubectl create`. + +```shell +kubectl create -f ./high-priority-pod.yml +``` + +Verify that "Used" stats for "high" priority quota, `pods-high`, has changed and that +the other two quotas are unchanged. + +```shell +kubectl describe quota +``` + +```shell +Name: pods-high +Namespace: default +Resource Used Hard +-------- ---- ---- +cpu 500m 1k +memory 10Gi 200Gi +pods 1 10 + + +Name: pods-low +Namespace: default +Resource Used Hard +-------- ---- ---- +cpu 0 5 +memory 0 10Gi +pods 0 10 + + +Name: pods-medium +Namespace: default +Resource Used Hard +-------- ---- ---- +cpu 0 10 +memory 0 20Gi +pods 0 10 +``` + +`scopeSelector` supports the following values in the `operator` field: + +* `In` +* `NotIn` +* `Exist` +* `DoesNotExist` + +## Requests vs Limits + +When allocating compute resources, each container may specify a request and a limit value for either CPU or memory. +The quota can be configured to quota either value. + +If the quota has a value specified for `requests.cpu` or `requests.memory`, then it requires that every incoming +container makes an explicit request for those resources. If the quota has a value specified for `limits.cpu` or `limits.memory`, +then it requires that every incoming container specifies an explicit limit for those resources. + +## Viewing and Setting Quotas + +Kubectl supports creating, updating, and viewing quotas: + +```shell +kubectl create namespace myspace +``` + +```shell +cat < compute-resources.yaml +apiVersion: v1 +kind: ResourceQuota +metadata: + name: compute-resources +spec: + hard: + pods: "4" + requests.cpu: "1" + requests.memory: 1Gi + limits.cpu: "2" + limits.memory: 2Gi + requests.nvidia.com/gpu: 4 +EOF +``` + +```shell +kubectl create -f ./compute-resources.yaml --namespace=myspace +``` + +```shell +cat < object-counts.yaml +apiVersion: v1 +kind: ResourceQuota +metadata: + name: object-counts +spec: + hard: + configmaps: "10" + persistentvolumeclaims: "4" + replicationcontrollers: "20" + secrets: "10" + services: "10" + services.loadbalancers: "2" +EOF +``` + +```shell +kubectl create -f ./object-counts.yaml --namespace=myspace +``` + +```shell +kubectl get quota --namespace=myspace +``` + +```shell +NAME AGE +compute-resources 30s +object-counts 32s +``` + +```shell +kubectl describe quota compute-resources --namespace=myspace +``` + +```shell +Name: compute-resources +Namespace: myspace +Resource Used Hard +-------- ---- ---- +limits.cpu 0 2 +limits.memory 0 2Gi +pods 0 4 +requests.cpu 0 1 +requests.memory 0 1Gi +requests.nvidia.com/gpu 0 4 +``` + +```shell +kubectl describe quota object-counts --namespace=myspace +``` + +```shell +Name: object-counts +Namespace: myspace +Resource Used Hard +-------- ---- ---- +configmaps 0 10 +persistentvolumeclaims 0 4 +replicationcontrollers 0 20 +secrets 1 10 +services 0 10 +services.loadbalancers 0 2 +``` + +Kubectl also supports object count quota for all standard namespaced resources +using the syntax `count/.`: + +```shell +kubectl create namespace myspace +``` + +```shell +kubectl create quota test --hard=count/deployments.extensions=2,count/replicasets.extensions=4,count/pods=3,count/secrets=4 --namespace=myspace +``` + +```shell +kubectl run nginx --image=nginx --replicas=2 --namespace=myspace +``` + +```shell +kubectl describe quota --namespace=myspace +``` + +```shell +Name: test +Namespace: myspace +Resource Used Hard +-------- ---- ---- +count/deployments.extensions 1 2 +count/pods 2 3 +count/replicasets.extensions 1 4 +count/secrets 1 4 +``` + +## Quota and Cluster Capacity + +`ResourceQuotas` are independent of the cluster capacity. They are +expressed in absolute units. So, if you add nodes to your cluster, this does *not* +automatically give each namespace the ability to consume more resources. + +Sometimes more complex policies may be desired, such as: + + - Proportionally divide total cluster resources among several teams. + - Allow each tenant to grow resource usage as needed, but have a generous + limit to prevent accidental resource exhaustion. + - Detect demand from one namespace, add nodes, and increase quota. + +Such policies could be implemented using `ResourceQuotas` as building blocks, by +writing a "controller" that watches the quota usage and adjusts the quota +hard limits of each namespace according to other signals. + +Note that resource quota divides up aggregate cluster resources, but it creates no +restrictions around nodes: pods from several namespaces may run on the same node. + +## Limit Priority Class consumption by default + +It may be desired that pods at a particular priority, eg. "cluster-services", should be allowed in a namespace, if and only if, a matching quota object exists. + +With this mechanism, operators will be able to restrict usage of certain high priority classes to a limited number of namespaces and not every namespace will be able to consume these priority classes by default. + +To enforce this, kube-apiserver flag `--admission-control-config-file` should be used to pass path to the following configuration file: + +```yaml +apiVersion: apiserver.k8s.io/v1alpha1 +kind: AdmissionConfiguration +plugins: +- name: "ResourceQuota" + configuration: + apiVersion: resourcequota.admission.k8s.io/v1beta1 + kind: Configuration + limitedResources: + - resource: pods + matchScopes: + - scopeName: PriorityClass + operator: In + values: ["cluster-services"] +``` + +Now, "cluster-services" pods will be allowed in only those namespaces where a quota object with a matching `scopeSelector` is present. +For example: +```yaml + scopeSelector: + matchExpressions: + - scopeName: PriorityClass + operator: In + values: ["cluster-services"] +``` + +See [LimitedResources](https://github.com/kubernetes/kubernetes/pull/36765) and [Quota support for priority class design doc](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/scheduling/pod-priority-resourcequota.md) for more information. + +## Example + +See a [detailed example for how to use resource quota](/docs/tasks/administer-cluster/quota-api-object/). + +{{% /capture %}} + +{{% capture whatsnext %}} + +See [ResourceQuota design doc](https://git.k8s.io/community/contributors/design-proposals/resource-management/admission_control_resource_quota.md) for more information. + +{{% /capture %}}