# Gameservers Scale
## The horizontal scaling feature of OpenKruiseGame

### Sequence of scale-down

OpenKruiseGame allows you to set the states of game servers. You can manually set the value of opsState or DeletionPriority for a game server. You can also use the service quality feature to automatically set the value of opsState or DeletionPriority for a game server. During scale-in, a proper GameServerSet workload is selected for scale-down based on the states of game servers. The scale-down rules are as follows:

1. Scale in game servers based on the opsState values. Scale in the game servers for which the opsState values are `WaitToBeDeleted`, `None`, `Allocated`, and `Maintaining` in sequence.

2. If two or more game servers have the same opsState value, game servers are scaled-down based on the values of DeletionPriority. The game server with the largest DeletionPriority value is deleted first.

3. If two or multiple game servers have the same opsState value and DeletionPriority value, the game server whose name contains the largest sequence number in the end is deleted first.

#### Examples

Deploy a game server with five replicas:

```bash
cat <<EOF | kubectl apply -f -
apiVersion: game.kruise.io/v1alpha1
kind: GameServerSet
metadata:
  name: minecraft
  namespace: default
spec:
  replicas: 5
  updateStrategy:
    rollingUpdate:
      podUpdatePolicy: InPlaceIfPossible
  gameServerTemplate:
    spec:
      containers:
        - image: registry.cn-hangzhou.aliyuncs.com/acs/minecraft-demo:1.12.2
          name: minecraft
EOF
```

Five game servers are generated:

```bash
kubectl get gs
NAME          STATE   OPSSTATE   DP    UP
minecraft-0   Ready   None       0     0
minecraft-1   Ready   None       0     0
minecraft-2   Ready   None       0     0
minecraft-3   Ready   None       0     0
minecraft-4   Ready   None       0     0
```

Set DeletionPriority to 10 for minecraft-2:

```bash
kubectl edit gs minecraft-2

...
spec:
  DeletionPriority: 10 # Change the value of DeletionPriority from the initial value 0 to 10.
  opsState: None
  updatePriority: 0
...
```

Manually perform scale-down to reduce the number of the game servers to 4:

```bash
kubectl scale gss minecraft --replicas=4
gameserverset.game.kruise.io/minecraft scale
```

The number of the game servers is changed to 4. The following example shows that minecraft-2 is deleted because it has the largest DeletionPriority value.

```bash
kubectl get gs
NAME          STATE      OPSSTATE   DP    UP
minecraft-0   Ready      None       0     0
minecraft-1   Ready      None       0     0
minecraft-2   Deleting   None       10    0
minecraft-3   Ready      None       0     0
minecraft-4   Ready      None       0     0

# After a while
...

kubectl get gs
NAME          STATE   OPSSTATE   DP    UP
minecraft-0   Ready   None       0     0
minecraft-1   Ready   None       0     0
minecraft-3   Ready   None       0     0
minecraft-4   Ready   None       0     0
```

Set opsState to WaitToBeDeleted for minecraft-3:

```bash
kubectl edit gs minecraft-3

...
spec:
  deletionPriority: 0
  opsState: WaitToBeDeleted # Change the value of opsState from the initial value None to WaitToBeDeleted.
  updatePriority: 0
...
```

Manually perform scale-down to reduce the number of the game servers to 3:

```bash
kubectl scale gss minecraft --replicas=3
gameserverset.game.kruise.io/minecraft scaled
```

The number of replicas for the game server is changed to 3. You can see that minecraft-3 is deleted because its opsState value is WaitToBeDeleted.

```bash
kubectl get gs
NAME          STATE      OPSSTATE          DP    UP
minecraft-0   Ready      None              0     0
minecraft-1   Ready      None              0     0
minecraft-3   Deleting   WaitToBeDeleted   0     0
minecraft-4   Ready      None              0     0

# After a while
...

kubectl get gs
NAME          STATE   OPSSTATE   DP    UP
minecraft-0   Ready   None       0     0
minecraft-1   Ready   None       0     0
minecraft-4   Ready   None       0     0
```

Manually perform scale-up and change the number of replicas for the game server back to 5:

```bash
kubectl scale gss minecraft --replicas=5
gameserverset.game.kruise.io/minecraft scaled
```

The number of replicas for the game server is changed back to 5. You can see that minecraft-2 and minecraft-3 are added for the game server.

```bash
kubectl get gs
NAME          STATE   OPSSTATE   DP    UP
minecraft-0   Ready   None       0     0
minecraft-1   Ready   None       0     0
minecraft-2   Ready   None       0     0
minecraft-3   Ready   None       0     0
minecraft-4   Ready   None       0     0
```

### GameServer ID Reserve

GameServerSet provides the `Spec.ReserveGameServerIds` field. Through this field, the user specifies the ID to delete the corresponding game service; or to avoid the generation of the game service corresponding to this serial number when creating a new game service.

For example, there exist 5 game suits under gss with IDs 0, 1, 2, 3 and 4. At this time, set `ReserveGameServerIds` and fill in 3 and 4, without changing the number of copies, gss will delete 3 and 4 and generate game suits of 5 and 6 at the same time, as follows:

```bash
kubectl edit gss minecraft
...
spec:
  reserveGameServerIds:
  - 3
  - 4
...

# After a while
kubectl get gs
NAME          STATE      OPSSTATE   DP    UP    AGE
minecraft-0   Ready      None       0     0     79s
minecraft-1   Ready      None       0     0     79s
minecraft-2   Ready      None       0     0     79s
minecraft-3   Deleting   None       0     0     78s
minecraft-4   Deleting   None       0     0     78s
minecraft-5   Ready      None       0     0     23s
minecraft-6   Ready      None       0     0     23s
```

If you fill in the `ReserveGameServerIds` field by adding 5 and 6 while decreasing the number of copies to 3, gss will delete the game service for 5 and 6, as follows:

```bash
kubectl edit gss minecraft
...
spec:
  replicas: 3
  reserveGameServerIds:
  - 3
  - 4
  - 5
  - 6
...

# After a while
kubectl get gs
NAME          STATE      OPSSTATE   DP    UP    AGE
minecraft-0   Ready      None       0     0     10m
minecraft-1   Ready      None       0     0     10m
minecraft-2   Ready      None       0     0     10m
minecraft-5   Deleting   None       0     0     9m55s
minecraft-6   Deleting   None       0     0     9m55s
```

**When downsizing, OKG will give priority to the game suits that are Reserved, and then follow the downsizing order mentioned above**

### Kill GameServer

OKG provides Kill mode to delete specified game server. Users only need to mark the OpsState of the game server they want to delete as "Kill".
Different from `Game Server ID Reserved`, in this mode, users do not need to manually adjust the replicas, OKG will automatically reduce replicas according to the number of GameServers whose OpsState is Kill; in addition, the deleted game server ID will not appear in the `ReserveGameServerIds` field(With General scaleDownStrategyType), which means that the game server with the corresponding serial number may be regenerated during scaling-up.

For example:
```bash
# There are 3 GameServers
kubectl get gs
NAME          STATE      OPSSTATE   DP    UP    AGE
minecraft-0   Ready      None       0     0     70s
minecraft-1   Ready      None       0     0     70s
minecraft-2   Ready      None       0     0     70s

# If you want to delete game server 1, just mark its OpsState as "Kill"
kubectl edit gs minecraft-1
...
spec:
  opsState: Kill
...

# minecraft-1 will be deleted, and the replicas of gss becomes 2
kubectl get gs
NAME          STATE      OPSSTATE   DP    UP    AGE
minecraft-0   Ready      None       0     0     78s
minecraft-1   Deleting   Kill       0     0     78s
minecraft-2   Ready      None       0     0     78s

kubectl get gs
NAME          STATE      OPSSTATE   DP    UP    AGE
minecraft-0   Ready      None       0     0     82s
minecraft-2   Ready      None       0     0     82s
```

### Strategy of scale-down

OKG provides two kinds of scale-down strategies: 1) General; 2) ReserveIds. You can set the corresponding strategy in `GameServerSet.Spec.ScaleStrategy`.

#### General

When the user does not configure the ScaleDownStrategyType field, General is the default configuration. The scale-down behavior is as described above.

#### ReserveIds

The user sets the ScaleDownStrategyType to `ReserveIds`, when the game service is scale-down, the deleted game service tail serial numbers will be recorded in reserveGameServerIds, and these tail serial numbers will not be used again when scale-up; if you want to use these tail serial numbers again, you only need to take them out from reserveGameServerIds and adjust the number of copies at the same time.

#### Examples

For example, there are 5 game services under gss, with IDs 0, 1, 2, 3, 4. Set `GameServerSet.Spec.ScaleStrategy.ScaleDownStrategyType` to `ReserveIds` and reduce the number of copies to 3.

```bash
kubectl edit gss minecraft
...
spec:
  replicas: 3
  scaleStrategy:
    scaleDownStrategyType: ReserveIds
...

# After a while
kubectl get gs
NAME          STATE      OPSSTATE   DP    UP    AGE
minecraft-0   Ready      None       0     0     10m
minecraft-1   Ready      None       0     0     10m
minecraft-2   Ready      None       0     0     10m
minecraft-3   Deleting   None       0     0     9m55s
minecraft-4   Deleting   None       0     0     9m55s
...

kubectl get gss minecraft -o yaml
spec:
  replicas: 3
  reserveGameServerIds:
  - 3
  - 4
  scaleStrategy:
  scaleDownStrategyType: ReserveIds
```

As you can see, the game services with serial numbers 3 and 4 are backfilled into the `reserveGameServerIds` field. At this point, if you wish to specify the expansion of game service number 4, remove 4 from the reserveGameServerIds and increase the number of copies to 4:

```bash
kubectl edit gss minecraft
...
spec:
  replicas: 4
  reserveGameServerIds:
  - 3
  scaleStrategy:
    scaleDownStrategyType: ReserveIds
...

# After a while

kubectl get gs
NAME          STATE   OPSSTATE   DP    UP    AGE
minecraft-0   Ready   None       0     0     17m
minecraft-1   Ready   None       0     0     17m
minecraft-2   Ready   None       0     0     17m
minecraft-4   Ready   None       0     0     6s
```

This function can scale up the game service with the specified serial number.

## Horizontal autoscale of gameserver

### Auto Scaling-down

Compared to stateless service types, game servers have higher requirements for automatic scaling, especially in terms of scaling down.

The differences between game servers become more and more obvious over time, and the precision requirements for scaling down are extremely high. Coarse-grained scaling mechanisms can easily cause negative effects such as player disconnections, resulting in huge losses for the business.

The horizontal scaling mechanism in native Kubernetes is shown in the following figure:

<img src={require('/static/img/kruisegame/user-manuals/autoscaling-k8s-en.png').default} style={{ height: '400px' , width: '700px'}} />

In the game scenario, its main problems are:

- At the pod level, it is unable to perceive the game server game status and therefore cannot set deletion priority based on game status.
- At the workload level, it cannot select scale-down objects based on game status.
- At the autoscaler level, it cannot accurately calculate the appropriate number of replicas based on the game server game status.

In this way, the automatic scaling mechanism based on native Kubernetes will cause two major problems in the game scenario:

- The number of scale-down is not accurate. It is easy to delete too many or too few game servers.
- The scale-down object is not accurate. It is easy to delete game servers with high game load levels.


The automatic scaling mechanism of OKG is shown in the following figure:

<img src={require('/static/img/kruisegame/user-manuals/autoscaling-okg-en.png').default} style={{ height: '400px' , width: '700px'}} />

- At the game server level, each game server can report its own status and expose whether it is in the WaitToBeDeleted state through custom service quality or external components.
- At the workload level, the GameServerSet can determine the scale-down object based on the business status reported by the game server. As described in [Game Server Horizontal Scaling](gameservers-scale.md), the game server in the WaitToBeDeleted state is the highest priority game server to be deleted during scale-down.
- At the autoscaler level, accurately calculate the number of game servers in the WaitToBeDeleted state, and use it as the scale-down quantity to avoid accidental deletion.

In this way, OKG's automatic scaler will only delete game servers in the WaitToBeDeleted state during the scale-down window, achieving targeted and precise scale-down.

**Usage Example**

_**Prerequisites: Install [KEDA](https://keda.sh/docs/2.10/deploy/) in the cluster.**_

Deploy the ScaledObject object to set the automatic scaling strategy. Refer to the [ScaledObject API](https://github.com/kedacore/keda/blob/main/apis/keda/v1alpha1/scaledobject_types.go) for the specific field meanings.

```yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: minecraft # Fill in the name of the corresponding GameServerSet
spec:
  scaleTargetRef:
    name: minecraft # Fill in the name of the corresponding GameServerSet
    apiVersion: game.kruise.io/v1alpha1 
    kind: GameServerSet
  pollingInterval: 30
  minReplicaCount: 0
  advanced:
    horizontalPodAutoscalerConfig: 
      behavior: # Inherit from HPA behavior, refer to https://kubernetes.io/zh-cn/docs/tasks/run-application/horizontal-pod-autoscale/#configurable-scaling-behavior
        scaleDown:
          stabilizationWindowSeconds: 45 # Set the scaling-down stabilization window time to 45 seconds
          policies:
            - type: Percent
              value: 100
              periodSeconds: 15
  triggers:
    - type: external
      metricType: AverageValue
      metadata:
        scalerAddress: kruise-game-external-scaler.kruise-game-system:6000

```

After deployment, change the opsState of the gs minecraft-0 to WaitToBeDeleted (see [Custom Service Quality](service-qualities.md) for automated setting of game server status).

```bash
kubectl edit gs minecraft-0

...
spec:
  deletionPriority: 0 
  opsState: WaitToBeDeleted # Set to None initially, and change it to WaitToBeDeleted
  updatePriority: 0
...

```

After the scale-down window period, the game server minecraft-0 is automatically deleted.

```bash
kubectl get gs
NAME          STATE      OPSSTATE          DP    UP
minecraft-0   Deleting   WaitToBeDeleted   0     0
minecraft-1   Ready      None              0     0
minecraft-2   Ready      None              0     0

# After a while


kubectl get gs
NAME          STATE   OPSSTATE   DP    UP
minecraft-1   Ready   None       0     0
minecraft-2   Ready   None       0     0
```

#### Set the maximum number of game servers whose opsState is None

OKG supports setting the maximum number of game servers. When the number of game servers with opsState None is greater than the set value, OKG will reduce Replicas so that the number of game servers with opsState None meets the set maximum number.

Considering the situation where the GameServer is idle and then reused, there are only two states of opsState: None and Allocated. None means that the game server is idle, and Allocated means that the game server is being used.
The horizontal scaling of the game server will be completely controlled by the minAvailable and maxAvailable parameters. When the number of None is less than minAvailable, OKG will automatically expand, and when the number of None is greater than maxAvailable, OKG will automatically shrink.

The maxAvailable parameter configuration method is as follows:

```yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: minecraft # Fill in the name of the corresponding GameServerSet
spec:
  scaleTargetRef:
    name: minecraft # Fill in the name of the corresponding GameServerSet
    apiVersion: game.kruise.io/v1alpha1
    kind: GameServerSet
  pollingInterval: 30
  minReplicaCount: 0
  advanced:
    horizontalPodAutoscalerConfig:
      behavior: # Inherit from HPA behavior, refer to https://kubernetes.io/zh-cn/docs/tasks/run-application/horizontal-pod-autoscale/#configurable-scaling-behavior
        scaleDown:
          stabilizationWindowSeconds: 30 # Set the scaling-down stabilization window time to 30 seconds
          policies:
            - type: Percent
              value: 100
              periodSeconds: 15
  triggers:
    - type: external
      metricType: AverageValue
      metadata:
        maxAvailable: "3" # set the maxAvailable of game servers whose opsState are None
        scalerAddress: kruise-game-external-scaler.kruise-game-system:6000
```

Initially deploy GameServerSet with 4 replicas. After the KEDA detection cycle, immediately scale down one game server.
At this time, the number of game servers with opsState of None is not greater than the set maxAvailable value, and the automatic scale-down is completed.

```bash
kubectl get gs
NAME          STATE   OPSSTATE   DP    UP   AGE
minecraft-0   Ready   None       0     0    10s
minecraft-1   Ready   None       0     0    10s
minecraft-2   Ready   None       0     0    10s
minecraft-3   Ready   None       0     0    10s

# After a while

kubectl get gs
NAME          STATE   OPSSTATE   DP    UP   AGE
minecraft-0   Ready   None       0     0    50s
minecraft-1   Ready   None       0     0    50s
minecraft-2   Ready   None       0     0    50s
```

### Auto Scaling-up

In addition to setting the automatic scaling policy, you can also set the automatic scaling policy.

#### Scaling with resource metrics or custom metrics

Native Kubernetes supports auto scaling-up using CPU utilization, and its complete yaml is as follows:

```yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: minecraft # Fill in the name of the corresponding GameServerSet
spec:
  scaleTargetRef:
    name: minecraft # Fill in the name of the corresponding GameServerSet
    apiVersion: game.kruise.io/v1alpha1
    kind: GameServerSet
  pollingInterval: 30
  minReplicaCount: 0
  advanced:
    horizontalPodAutoscalerConfig:
      behavior: # Inherit from HPA behavior, refer to https://kubernetes.io/zh-cn/docs/tasks/run-application/horizontal-pod-autoscale/#configurable-scaling-behavior
        scaleDown:
          stabilizationWindowSeconds: 45 # Set the scaling-down stabilization window time to 45 seconds
          policies:
            - type: Percent
              value: 100
              periodSeconds: 15
  triggers:
    - type: external
      metricType: AverageValue
      metadata:
        scalerAddress: kruise-game-external-scaler.kruise-game-system:6000
    - type: cpu
      metricType: Utilization    # Allowed types are 'Utilization' or 'AverageValue'
      metadata:
        value: "50"
```

Pressure testing of the gameserver, you can see that the gameserver began to scale-up

```bash
kubectl get gss
NAME        DESIRED   CURRENT   UPDATED   READY   MAINTAINING   WAITTOBEDELETED   AGE
minecraft   5         5         5         0       0             0                 7s

# After a while

kubectl get gss
NAME        DESIRED   CURRENT   UPDATED   READY   MAINTAINING   WAITTOBEDELETED   AGE
minecraft   20        20        20        20      0             0                 137s
```

#### Set the minimum number of game servers whose opsState is None

OKG supports setting the minimum number of game servers. When the current number of game servers whose opsState is None is less than the set value, OKG will automatically expand new game servers so that the number of game servers whose opsState is None meets the set minimum number.

The configuration method is as follows. In this example, the minimum number of game servers with opsState set to None is 3:

```yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: minecraft # Fill in the name of the corresponding GameServerSet
spec:
  scaleTargetRef:
    name: minecraft # Fill in the name of the corresponding GameServerSet
    apiVersion: game.kruise.io/v1alpha1
    kind: GameServerSet
  pollingInterval: 30
  minReplicaCount: 0
  advanced:
    horizontalPodAutoscalerConfig:
      behavior: # Inherit from HPA behavior, refer to https://kubernetes.io/zh-cn/docs/tasks/run-application/horizontal-pod-autoscale/#configurable-scaling-behavior
        scaleDown:
          stabilizationWindowSeconds: 45 # Set the scaling-down stabilization window time to 45 seconds
          policies:
            - type: Percent
              value: 100
              periodSeconds: 15
  triggers:
    - type: external
      metricType: AverageValue
      metadata:
        minAvailable: "3" # 设置opsState为None的游戏服的最小个数
        scalerAddress: kruise-game-external-scaler.kruise-game-system:6000
```

First apply a GameServerSet with 1 replicas, after the KEDA detection cycle, immediately scale up two new game servers. At this time, the number of game servers whose opsState is None is not less than the minAvailable value, and scale-up process is completed.

```bash
kubectl get gs
NAME          STATE   OPSSTATE   DP    UP   AGE
minecraft-0   Ready   None       0     0    7s

# After a while

kubectl get gs
NAME          STATE   OPSSTATE   DP    UP   AGE
minecraft-0   Ready   None       0     0    20s
minecraft-1   Ready   None       0     0    5s
minecraft-2   Ready   None       0     0    5s
```

### Other Settings

Kubernetes has a certain tolerance for automatic scaling behavior, which is determined by the kube-controller-manager parameter --horizontal-pod-autoscaler-tolerance, and the default is 0.1, which means that the difference between the desired replicas and the current replicas is 10% No expansion or contraction will be triggered when the value is within.
If more accurate auto-scaling is achieved, this parameter can be lowered. For example, when it is set to 0.0, OKG will scale down all GameServers with WaitToBeDeleted opsState.