kubernetes
/
autoscaler
mirror of https://github.com/kubernetes/autoscaler.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
autoscaler/cluster-autoscaler/cloudprovider/azure
History
Denton24646 a1095407ce add NV6Promo to azure_instance_types.go 2019-09-12 09:33:56 -04:00
..
examples Fix typo in autoscaler azure example 2019-07-18 10:23:09 +02:00
OWNERS Add reviewers 2018-02-14 20:30:53 +08:00
README.md Merge pull request #2292 from nilo19/t-qini-update_readme_scale_boundary 2019-09-03 19:42:59 -07:00
azure_agent_pool.go Improve autoscaler on Azure README 2019-05-21 16:21:21 -07:00
azure_cache.go Convert resource group name in Azure providerID to lower case 2019-03-08 10:47:27 +08:00
azure_client.go Merge branch 'master' into crc-fix-error-format 2019-02-01 17:22:57 +01:00
azure_cloud_provider.go Ignore nodes that have no ProviderID. (kubernetes#2236) 2019-08-04 15:20:21 +09:00
azure_cloud_provider_test.go Upgrade Azure compute APIs 2018-11-19 16:51:03 +08:00
azure_container_service_pool.go Change the rule when setting the nodes' size after scaling. 2019-08-27 11:17:36 +08:00
azure_fakes.go Look up Azure instance types case-insensitively 2019-05-15 17:49:49 -06:00
azure_instance_types.go add NV6Promo to azure_instance_types.go 2019-09-12 09:33:56 -04:00
azure_manager.go Use k8s.io/klog instead github.com/golang/glog 2018-11-26 17:30:31 +01:00
azure_scale_set.go Fix race conditions when checking the VMSS size 2019-07-08 16:45:42 +08:00
azure_scale_set_test.go Add cache for vmss VM 2019-07-03 22:46:03 +08:00
azure_util.go Refactor the corresponding code. 2019-07-22 08:58:51 +08:00
azure_util_test.go Fix Azure client requests stuck issues on http.StatusTooManyRequests 2019-07-03 22:47:23 +08:00

README.md

Cluster Autoscaler on Azure

The cluster autoscaler on Azure dynamically scales Kubernetes worker nodes. It runs as a deployment in your cluster.

This README will help you get cluster autoscaler running on your Azure Kubernetes cluster.

Kubernetes Version

Kubernetes v1.10.x or later is required to use cluster autoscaler on Azure. See the "Releases" section in the README for more information.

CA Version

Cluster autoscaler v1.2.x or later is required for Azure. See the "Releases" section in the README for more information.

NOTE: In the deployment manifests referenced below, be sure to replace the {{ ca_version }} placeholder with an actual release, such as v1.14.2.

Permissions

Get Azure credentials by running the following Azure CLI command:

# replace <subscription-id> with yours.
az ad sp create-for-rbac --role="Contributor" --scopes="/subscriptions/<subscription-id>" --output json

This will create a new service principal with "Contributor" role scoped to your subscription. Save the JSON output, because it will be needed to configure the cluster autoscaler deployment in the next step.

Deployment manifests

Cluster autoscaler supports four Kubernetes cluster options on Azure:

  • vmss: Autoscale VMSS instances by setting the Azure cloud provider's vmType parameter to vmss or to an empty string. This supports clusters deployed with aks-engine.
  • standard: Autoscale VMAS instances by setting the Azure cloud provider's vmType parameter to standard. This supports clusters deployed with aks-engine.
  • aks: Supports an Azure Kubernetes Service (AKS) cluster.
  • DEPRECATED acs: Supports an Azure Container Service (ACS) cluster.

NOTE: only the vmss option supports scaling down to zero nodes.

VMSS deployment

Prerequisites:

  • Get Azure credentials from the Permissions step above.
  • Get the name of the VM scale set associated with the cluster's node pool. You can find this in the Azure Portal or with the az vmss list command.

Make a copy of cluster-autoscaler-vmss.yaml. Fill in the placeholder values for the cluster-autoscaler-azure secret data by base64-encoding each of your Azure credential fields.

  • ClientID: <base64-encoded-client-id>
  • ClientSecret: <base64-encoded-client-secret>
  • ResourceGroup: <base64-encoded-resource-group>
  • SubscriptionID: <base64-encoded-subscription-id>
  • TenantID: <base64-encoded-tenant-id>

NOTE: Use a command such as echo $CLIENT_ID | base64 to encode each of the fields above.

In the cluster-autoscaler spec, find the image: field and replace {{ ca_version }} with a specific cluster autoscaler release.

Below that, in the command: section, update the --nodes= arguments to reference your node limits and VMSS name. For example, if node pool "k8s-nodepool-1-vmss" should scale from 1 to 10 nodes:

        - --nodes=1:10:k8s-nodepool-1-vmss

or to autoscale multiple VM scale sets:

        - --nodes=1:10:k8s-nodepool-1-vmss
        - --nodes=1:10:k8s-nodepool-2-vmss

Note that it doesn't mean the number of nodes in nodepool is restricted in the range from 1 to 10. It means when ca is downscaling (upscaling) the nodepool, it will never break the limit of 1 (10). If the current node pool size is lower than the specified minimum or greater than the specified maximum when you enable autoscaling, the autoscaler waits to take effect until a new node is needed in the node pool or until a node can be safely deleted from the node pool.

To allow scaling similar node pools simultaneously, or when using separate node groups per zone and to keep nodes balanced across zones, use the --balance-similar-node-groups flag (default false). Add it to the command section to enable it:

        - --balance-similar-node-groups=true

See the FAQ for more details.

Save the updated deployment manifest, then deploy cluster-autoscaler by running:

kubectl create -f cluster-autoscaler-vmss.yaml

To run a cluster autoscaler pod on a master node, the deployment should tolerate the master taint, and nodeSelector should be used to schedule pods. Use cluster-autoscaler-vmss-master.yaml in this case.

To run a cluster autoscaler pod with Azure managed service identity (MSI), use cluster-autoscaler-vmss-msi.yaml instead.

Standard deployment

Prerequisites:

  • Get Azure credentials from the Permissions step above.
  • Get the name of the initial Azure deployment resource for the cluster. You can find this in the Azure Portal or with the az deployment list command. If there are multiple deployments, get the name of the first one.

Make a copy of cluster-autoscaler-standard-master.yaml. Fill in the placeholder values for the cluster-autoscaler-azure secret data by base64-encoding each of your Azure credential fields.

  • ClientID: <base64-encoded-client-id>
  • ClientSecret: <base64-encoded-client-secret>
  • ResourceGroup: <base64-encoded-resource-group>
  • SubscriptionID: <base64-encoded-subscription-id>
  • TenantID: <base64-encoded-tenant-id>
  • Deployment: <base64-encoded-azure-initial-deployment-name>

NOTE: Use a command such as echo $CLIENT_ID | base64 to encode each of the fields above.

In the cluster-autoscaler spec, find the image: field and replace {{ ca_version }} with a specific cluster autoscaler release.

Below that, in the command: section, update the --nodes= arguments to reference your node limits and node pool name. For example, if node pool "k8s-nodepool-1" should scale from 1 to 10 nodes:

        - --nodes=1:10:k8s-nodepool-1

or to autoscale multiple VM scale sets:

        - --nodes=1:10:k8s-nodepool-1
        - --nodes=1:10:k8s-nodepool-2

Create the Azure deploy parameters secret cluster-autoscaler-azure-deploy-parameters by running:

kubectl -n kube-system create secret generic cluster-autoscaler-azure-deploy-parameters --from-file=deploy-parameters=./_output/<your-output-path>/azuredeploy.parameters.json

Then deploy cluster-autoscaler by running:

kubectl create -f cluster-autoscaler-standard-master.yaml

To run a cluster autoscaler pod with Azure managed service identity (MSI), use cluster-autoscaler-standard-msi.yaml instead.

WARNING: Cluster autoscaler depends on user-provided deployment parameters to provision new nodes. After upgrading your Kubernetes cluster, cluster autoscaler must also be redeployed with new parameters to prevent provisioning nodes with an old version.

AKS deployment

Autoscaling VM scale sets with AKS is supported for Kubernetes v1.12.4 and later. The option to enable cluster autoscaler is available in the Azure Portal or with the Azure CLI:

az aks create \
  --resource-group myResourceGroup \
  --name myAKSCluster \
  --kubernetes-version 1.13.5 \
  --node-count 1 \
  --enable-vmss \
  --enable-cluster-autoscaler \
  --min-count 1 \
  --max-count 3

Please see the AKS autoscaler documentation for details.

ACS deployment

NOTE: ACS will retire on January 31, 2020.

Prerequisites:

  • Get Azure credentials from the Permissions step above.
  • Get the cluster name with the az acs list command.
  • Get the name of a node pool from the value of the label agentpool
kubectl get nodes --show-labels

Make a copy of cluster-autoscaler-containerservice. Fill in the placeholder values for the cluster-autoscaler-azure secret data by base64-encoding each of your Azure credential fields.

  • ClientID: <base64-encoded-client-id>
  • ClientSecret: <base64-encoded-client-secret>
  • ResourceGroup: <base64-encoded-resource-group> (Note: ResourceGroup is case-sensitive)
  • SubscriptionID: <base64-encoded-subscription-id>
  • TenantID: <base64-encoded-tenant-id>
  • ClusterName: <base64-encoded-clustername>
  • NodeResourceGroup: <base64-encoded-node-resource-group> (Note: node resource group is not resource group and can be obtained in the corresponding label of the nodepool)

NOTE: Use a command such as echo $CLIENT_ID | base64 to encode each of the fields above.

In the cluster-autoscaler spec, find the image: field and replace {{ ca_version }} with a specific cluster autoscaler release.

Below that, in the command: section, update the --nodes= arguments to reference your node limits and node pool name. For example, if node pool "k8s-nodepool-1" should scale from 1 to 10 nodes:

        - --nodes=1:10:k8s-nodepool-1

or to autoscale multiple VM scale sets:

        - --nodes=1:10:k8s-nodepool-1
        - --nodes=1:10:k8s-nodepool-2

Then deploy cluster-autoscaler by running

kubectl create -f cluster-autoscaler-containerservice.yaml