9bb2fd15d7
Add TODO
2020-02-04 20:51:42 +01:00
69800ab176
Simulate scheduling of pods waiting for preemption in ClusterSnapshot
2020-02-04 20:51:37 +01:00
d9891ae3ad
Simplify PodListProcessor interface
2020-02-04 20:51:35 +01:00
7e62105cb9
Add upcoming nodes to ClusterSnapshot
2020-02-04 20:51:31 +01:00
83d1c4ff8a
Add GetAllPods and GetAllNodes to ClusterSnapshot
2020-02-04 20:51:30 +01:00
fa2c6e4d9e
Propagate cluster state to ClusterSnapshot
2020-02-04 20:51:27 +01:00
036103c553
Add ClusterSnapshot to AutoscalingContext
2020-02-04 20:51:26 +01:00
373c558303
Extract PredicateChecker interface
2020-02-04 20:51:18 +01:00
b01f2fca8f
Drop ConfigurePredicateCheckerForLoop
2020-02-04 20:51:14 +01:00
68433abb7c
Remove duplicate comments
2019-12-28 01:06:22 +08:00
f6ed9c114a
Merge pull request #2588 from losipiuk/lo/snapshot
...
Snapshot cluster state for scheduler every loop
2019-11-28 05:25:03 -08:00
b67854e800
Snapshot cluster state for scheduler every loop
...
Change-Id: If9d162b83ccc914fe1b02e4689bfe1f4b264407f
2019-11-28 14:02:08 +01:00
17a7bc5164
Ignore NominatedNodeName on Pod if node is gone
...
Change-Id: I4a119f46e55ca2223f9f0fdd3e75ce3f279e293b
2019-11-27 20:26:00 +01:00
910e75365c
remove temporary nodes logic
2019-11-12 11:58:29 +01:00
7c9d6e3518
Do not assign return values to variables when not needed
2019-10-25 19:28:00 +08:00
7f083d2393
Move core/utils.go to separate package and split into multiple files
2019-10-22 14:23:40 +02:00
288d4107b2
Rename GetCreatedNodesWithOutOfResourcesErrors to GetCreatedNodesWithErrors
2019-10-14 10:56:56 +02:00
474eef6d47
Invalidate node instances cache after deleting unregistered nodes
2019-09-30 14:29:02 +02:00
7c17d52ec8
Invalidate node instances cache after deleting failed nodes
2019-09-30 13:56:33 +02:00
791f0d8355
Merge pull request #2281 from DataDog/JulienBalestra/mig-block
...
cluster-autoscaler: blocked if an instance is detached from MIG
2019-09-11 05:03:22 -07:00
3441f616e1
cluster-autoscaler/skip-node: unblock cluster autoscaler when having a single nodegroup for node error
...
Signed-off-by: Julien Balestra <julien.balestra@datadoghq.com>
2019-09-11 13:40:23 +02:00
839cdaaa09
Stop disabling Cluster Autoscaler when there is no ready nodes.
2019-09-06 14:45:34 +02:00
6d707a08ac
cluster-autoscaler/metrics: expose the scale down cooldown
...
Signed-off-by: Julien Balestra <julien.balestra@datadoghq.com>
2019-08-27 18:12:33 +02:00
9aac43e237
Merge pull request #2235 from piontec/fix/aws_spots_squashed
...
correctly handle lack of capacity of AWS spot ASGs
2019-08-19 04:27:30 -07:00
3f0a5fa3c2
Merge pull request #2233 from vivekbagade/surge
...
Adding ScaleDownNodeProcessor
2019-08-19 03:59:32 -07:00
8d9b81caaa
correctly handle lack of capacity of AWS spot ASGs
2019-08-19 12:43:53 +02:00
dc64d0aab2
Adding ScaleDownNodeProcessor
2019-08-12 20:19:55 +02:00
44ae89dd09
Communicate the result of RemoveUnneededNodeGroups to ScaleDownStatusProcessor
2019-08-12 17:03:51 +02:00
f7c563ab06
Modify the code as the simple solution proposed by MaciekPytel.
2019-07-18 23:58:05 +08:00
622a838c2c
Modify nodal similarity rules.
2019-07-09 16:04:40 +08:00
d45fee06da
Ensure upcoming nodes are different
2019-07-02 16:52:19 +08:00
478660a6bb
fix error
2019-06-28 18:49:58 +08:00
90aa28a077
Move pod packing in upcoming nodes to RunOnce from Estimator for performance improvements
2019-06-19 14:48:47 +02:00
0bcf5315a7
Do not fail loop iteration if unregistered nodes cannot be removed
...
The mechanism of unregistered nodes removal is not the first
responsibility of Cluster Autoscaler. We do not want to renderi CA
unsable (disable scale-up and scale-down) if removing unregistered nodes
cannot be done for prolonged period of time.
2019-06-10 13:45:54 +02:00
bb382f47f9
Retain information about scale-up failures in CSR
...
This will provide the AutoscalingStatusProcessor with information
about failed scale-ups.
2019-06-05 16:53:30 +02:00
4831d76288
Cache cloud provider node instances in cluster state.
2019-05-31 10:11:51 +02:00
cb4e60f8d4
Merge pull request #2031 from krzysztof-jastrzebski/master
...
Add functionality which delays node deletion to let other components prepare for deletion.
2019-05-20 00:57:13 -07:00
8d2ec08b2c
Merge pull request #2015 from losipiuk/lo/pass-via-context
...
Add methods for passing arbitrary object via autoscaling context
2019-05-17 08:12:07 -07:00
e76558c65f
Add methods for passing arbitrary object via autoscaling context
...
Change-Id: I066e58010a0aef4989bfc1f73b90bc69c773b26e
2019-05-17 16:38:12 +02:00
4247c8b032
Implement functionality which delays node deletion when node has
...
annotation with prefix
'delay-deletion.cluster-autoscaler.kubernetes.io/'.
2019-05-17 16:06:17 +02:00
92ea680f1a
Implement an --ignore-taint flag
...
This change adds support for a user to specify taints to ignore when
considering a node as a template for a node group.
2019-05-14 10:22:59 -07:00
c9811e87b4
Include pods with NominatedNodeName in scheduledPods list used for scale-up considerations
...
Change-Id: Ie4c095b30bf0cd1f160f1ac4b8c1fcb8c0524096
2019-04-15 16:59:13 +02:00
db4c6f1133
Migrate filter out schedulabe to PodListProcessor
2019-04-15 16:59:13 +02:00
5c09c50774
Pass ready nodes list to PodListProcessor
2019-04-15 16:59:13 +02:00
c6115b826e
Define ProcessorCallbacks interface
2019-04-15 16:59:13 +02:00
90666881d3
Move GPULabel and GPUTypes to cloud provider
2019-03-25 13:03:01 -07:00
2474dc2fd5
Call CloudProvider.Refresh before getNodeInfosForGroups
...
We need to call refresh before getNodeInfosForGroups. If we have
stale state getNodeInfosForGroups may fail and we will end up in infinite crash looping.
2019-03-12 12:07:49 +01:00
62a28f3005
Soft taint when there are no candidates
2019-03-11 14:05:09 +01:00
91b7bc08a1
Fixing minor error handling bug in static autoscaler
2019-03-07 15:16:27 -08:00
a824e87957
Only soft taint nodes if there's no scale down to do
2019-02-25 17:11:15 +01:00
128729bae9
Move schedulercache to package nodeinfo
2019-02-21 12:41:08 +08:00
d969baff22
Cache exemplar ready node for each node group
2019-02-11 17:40:58 +01:00
c6b87841ce
Added a new method that uses pod packing to filter schedulable pods
...
filterOutSchedulableByPacking is an alternative to the older
filterOutSchedulable. filterOutSchedulableByPacking sorts pods in
unschedulableCandidates by priority and filters out pods that can be
scheduled on free capacity on existing nodes. It uses a basic packing
approach to do this. Pods with nominatedNodeName set are always
filtered out.
filterOutSchedulableByPacking is set to be used by default, but, this
can be toggled off by setting filter-out-schedulable-pods-uses-packing
flag to false, which would then activate the older and more lenient
filterOutSchedulable(now called filterOutSchedulableSimple).
Added test cases for both methods.
2019-01-25 16:09:51 +05:30
79ef3a6940
unexporting methods in utils.go
2019-01-25 00:06:03 +05:30
0c64e0932a
Tainting unneeded nodes as PreferNoSchedule
2019-01-21 13:06:50 +01:00
85a83b62bd
Pass nodeGroup->NodeInfo map to ClusterStateRegistry
...
Change-Id: Ie2a51694b5731b39c8a4135355a3b4c832c26801
2019-01-08 15:52:00 +01:00
ab7f1e69be
Merge pull request #1464 from losipiuk/lo/stockouts2
...
Better quota-exceeded/stockout handling
2018-12-31 05:28:08 -08:00
2fbae197f4
Handle possible stockout/quota scale-up errors
2018-12-28 17:17:07 +01:00
60babe7158
Use kubernetes lister for daemonset instead of custom one
...
Also migrate to using apps/v1.DaemonSet instead of old
extensions/v1beta1.
2018-12-28 13:55:41 +01:00
d0dd00c602
Fix logged error in static autoscaler
2018-12-04 16:59:57 +01:00
016bf7fc2c
Use k8s.io/klog instead github.com/golang/glog
2018-11-26 17:30:31 +01:00
5962354c81
Inject Backoff instance to ClusterStateRegistry on creation
2018-11-13 14:25:16 +01:00
bf6ff4be8e
Clean up estimators
2018-11-06 14:15:42 +01:00
8179e4e716
Refactor the scale-(up|down) status processors so that they have more info available
...
Replace the simple boolean ScaledUp property of ScaleUpStatus with a more
comprehensive ScaleUpResult. Add more possible values to ScaleDownResult.
Refactor the processors execution so that they are always executed every
iteration, even if RunOnce exits earlier.
2018-09-20 17:12:02 +02:00
88d857222d
Renamed one more variable for consistency
...
Change-Id: Idf42fd58089a1e75f3291ab7cc583735c68735f2
2018-09-17 14:08:10 -04:00
56b5456269
Fixing nits: renamed newPodScaleUpBuffer -> newPodScaleUpDelay, deleted redundant comment
...
Change-Id: I7969194d8e07e2fb34029d0d7990341c891d0623
2018-09-17 10:38:28 -04:00
33b93cbc5f
Add configurable delay for pod age before considering for scale-up
...
- This is intended to address the issue described in https://github.com/kubernetes/autoscaler/issues/923
- the delay is configurable via a CLI option
- in production (on AWS) we set this to a value of 2m
- the delay could possibly be set as low as 30s and still be effective depending on your workload and environment
- the default of 0 for the CLI option results in no change to the CA's behavior from defaults.
Change-Id: I7e3f36bb48641faaf8a392cca01a12b07fb0ee35
2018-09-14 13:55:09 -04:00
71111da20c
Add a scale down status processor, refactor so that there's more scale down info available to it
2018-09-12 14:52:20 +02:00
054f0b3b90
Add AutoscalingStatusProcessor
2018-08-07 14:47:06 +02:00
90e8a7a2d9
Move initializing defaults out of main
2018-08-02 14:04:03 +02:00
6f9b6f8290
Move ListerRegistry to context
2018-07-26 13:31:49 +02:00
07e52e6c79
Move creating cloud provider out of context
2018-07-25 13:43:47 +02:00
0976d2aa07
Move autoscaling options out of static
2018-07-25 10:52:37 +02:00
6b94d7172d
Move AutoscalingOptions to config/static
2018-07-23 15:52:27 +02:00
2df2568841
Move removing unneeded autoprovisioned node groups to node group manager
2018-06-22 14:26:12 +02:00
b8ae6df5d3
Add post scale up status processor.
2018-06-06 13:34:49 +02:00
856855987b
Move some GKE-specific logic outside core
...
No change in actual logic being executed. Added a new
NodeGroupListProcessor interface to encapsulate the existing logic.
Moved PodListProcessor and refactor how it's passed around
to make it consistent and easy to add similar interfaces.
2018-05-29 12:57:19 +02:00
5faa41e683
Move PodListProcessor to new directory
...
It's not really a util and with more processors
coming it makes more sense to keep them in dedicated place.
2018-05-29 12:00:47 +02:00
4c710950de
Move ClusterStateRegistry to StaticAutoscaler
...
AutoscalingContext is basically a configuration and few static helpers
and API handles.
ClusterStateRegistry is state and thus moved to other state-keeping
objects.
2018-05-24 13:03:01 +02:00
5bfab7d9b2
Return value moved to the caller
2018-05-18 14:59:15 +02:00
fa6f25a70a
Extract ClusterStateRegistry update with its soft dependency
2018-05-18 10:25:15 +02:00
dc34b43a40
Extract another tiny method
2018-05-18 10:10:51 +02:00
34f6a45a04
Extract method to hide a tiny bit of complexity
2018-05-18 10:01:52 +02:00
d7dc3616f7
Merge pull request #868 from kgolab/kg-clean-up-010
...
Move metrics update to proper place
2018-05-17 14:52:18 +02:00
e31bf0bb58
Move metrics.Autoscaling after all Node-level operations & checks
2018-05-17 14:37:43 +02:00
3b6cfc7c2b
Merge pull request #870 from kgolab/kg-clean-up-012
...
Set lastScaleDownFailTime properly
2018-05-17 12:09:15 +02:00
444201d1e7
Merge pull request #871 from kgolab/kg-clean-up-013
...
Extract duplicate code into a single method
2018-05-17 11:49:49 +02:00
400147a075
Extract duplicate code into a single method
2018-05-17 10:01:04 +02:00
b8cbdf4178
Set lastScaleDownFailTime properly - the ScaleDownError check was unreachable
2018-05-17 09:50:22 +02:00
38a5951e22
Check glog.V once
2018-05-17 09:47:52 +02:00
ccca078a2b
Move metrics update to proper place
2018-05-17 09:46:25 +02:00
bc39d4dcd5
Merge pull request #842 from kgolab/kg-clean-up-008
...
Merge two variables into one.
2018-05-14 10:54:43 +02:00
b52ec59b05
Fix cleaning up taints
2018-05-11 12:00:48 +02:00
f1f92f065e
Merge two variables into one.
2018-05-10 14:32:37 +02:00
ae203ed517
Removed unused CloudProvider() method.
2018-05-08 11:23:55 +02:00
854fcc1ff8
Remove implementation details (CleanUp) from the interface.
...
The CleanUp method is instead called directly from the implementation,
when required.
Test updated in a quick way since the mock we're using does not support
AtLeast(1) - thus Times(2).
2018-05-07 15:24:14 +02:00
88b769b324
Refactor cluster autoscaler builder and add pod list processor.
2018-04-26 12:37:51 +02:00
7e1353a865
Ignore TPU resource in simulations
2018-04-11 12:26:22 +02:00
abbc45da2e
Delay scale-up including GPU request
...
Nodes with GPU are expensive and it's likely a bunch of pods
using them will be created in a batch. In this case we can
wait a bit for all pods to be created to make more efficient
scale-up decision.
2018-03-02 15:55:04 +01:00
bf59e3daa5
Typo fix unneded->[unneeded] ( #623 )
...
* Update clusterstate.md
* Update scale_down.go
* Update static_autoscaler.go
2018-02-07 17:36:58 +01:00
346a5c26a9
Remove old unregistered nodes before checking cluster healthiness
2018-02-01 16:34:50 +01:00
b17b6c3ec5
Wait before publishing no nodes ready after start
2018-01-16 19:04:38 +01:00
27efa05b1d
Publish ClusterUnhealthy events
2018-01-16 16:56:36 +01:00
1b728d411b
Publish status and metrics for empty cluster
2018-01-16 16:07:29 +01:00
15b10c8f67
Skip iteration if pending pods are too new
2017-12-28 16:55:44 +01:00
ded016dfd8
Merge pull request #461 from MaciekPytel/gpu_unready_fix
...
Consider GPU nodes unready until allocatable GPU is > 0
2017-11-13 15:29:27 +01:00
d81dca5991
Mark nodes with uninitialized GPUs as unready
2017-11-10 17:56:10 +01:00
439fd3c9ec
Merge pull request #411 from krzysztof-jastrzebski/priority
...
Adds priority preemption support to cluster autoscaler.
2017-11-08 09:09:26 +01:00
2b28ac1a04
Add a workaround for scaling of VMs with GPUs
...
When a machine with GPU becomes ready it can take
up to 15 minutes before it reports that GPU is allocatable.
This can cause Cluster Autoscaler to trigger a second
unnecessary scale up.
The workaround sets allocatable to capacity for GPU so that
a node that waits for GPUs to become ready to use will be
considered as a place where pods requesting GPUs can be
scheduled.
2017-11-06 16:04:22 +01:00
4104a91991
more spelling fixes
2017-11-02 14:21:36 -07:00
9c2ebccbfe
Write events when autoprovisioned nodegroup is created / deleted
2017-10-25 17:39:30 +02:00
07511f444a
Add Refresh method to cloud provider
...
This can be used to dynamically update cloud provider
config (in particular list of managed NodeGroups and their
min/max constraints).
Add GKE implementation.
2017-10-24 18:36:29 +02:00
d9c00e5ce1
Adds priority preemption support to cluster autoscaler.
2017-10-23 09:54:56 +02:00
098ebbee09
Log event when removing unregistered node
2017-09-22 22:48:07 +02:00
5e05c84cf0
Add metric counting failed scale-ups
...
A minor refactor was required to avoid cyclic imports
2017-09-22 18:12:50 +02:00
63310ef41a
Introduce new flags to control scale down behavior: scale-down-delay-after-delete and scale-down-delay-after-failure, replacing scale-down-trial-interval. scale-down-delay-after-add replaces scale-down-delay
2017-09-18 17:09:44 -07:00
303f86c163
Merge pull request #336 from electronicarts/feature/matt/unneeded-check-fix
...
Move calculateUnneededOnly check after unneeded calculations
2017-09-13 11:14:51 +02:00
d8db14701e
Core/static_autoscaler_test.go unit tests.
2017-09-13 09:52:07 +02:00
43943cdeb4
Move calculateUnneededOnly check after unneeded calculations, add log message to main loop start
2017-09-12 21:38:29 -07:00
0aec68a46d
Core/static_autoscaler.go unit tests. Current time usage refactoring.
2017-09-11 15:07:21 +02:00
bcc8cded64
Clean up empty autoprovisioned node groups
2017-09-04 13:53:07 +02:00
69c5ea03ce
Disable MatchInterPodAffinity if there are no pods using affinity
2017-08-30 16:18:31 +02:00
6ad7ca21e8
Merge pull request #265 from MaciekPytel/ignore_unneded_if_min_size
...
Skip nodes in min-sized groups in scale-down simulation
2017-08-28 19:40:53 +05:30
9e2c76551f
Merge pull request #263 from mwielgus/delete-in-goroutine
...
Run node drain/delete in a separate goroutine
2017-08-28 19:39:57 +05:30
2f6dd8aefc
Skip nodes in min-sized groups in scale-down simulation
...
Currently we track if those nodes can be removed and only
skip them at the execution step. Since checking if node is
unneeded is pretty expensive it's better to filter them out
early.
2017-08-28 15:48:41 +02:00
718e5db78e
Run node drain/delete in a separate goroutine
2017-08-28 12:12:31 +02:00
71b4ca5461
Dont block stale downs if no nodes can be removed
2017-08-26 16:29:50 +02:00
edeb522274
Add measuring of FilterOutSchedulable
2017-08-22 18:36:13 +02:00
43c9b6b06b
Add cleaner function labels for metrics exporting.
2017-08-22 16:09:42 +02:00
14df1b808b
Drill down scale down metrics
...
Split scale down duration into three parts:
1. Find nodes to remove
2. Node deletion
3. Misc operations
2017-08-18 14:17:02 +02:00
95b5b4be94
Remove --verify-unschedulabe-pods flag
...
This flag was true in default setups for every platform,
we haven't heard about any user changing it to false and
after removing check on PodScheduled condition setting it
to false would basically break CA.
2017-08-16 17:31:59 +02:00
ef1241b3c6
Remove checking and resetting PodSchedulable condition
...
The performance cost was too high and the pods should
be filtered out by follow up checks anyway.
Check out https://github.com/kubernetes/autoscaler/issues/187
for details.
2017-08-16 17:30:11 +02:00
9116e4c08c
Compilation fix for CA after godeps update
2017-08-11 17:56:47 +02:00
902d2414b7
Fixed typoes of name 'Kubernetes'
2017-08-03 14:20:23 +12:00
55d750196c
Add a flag to turn off pod status condition reseting for performance tests
2017-07-24 15:53:45 +02:00
2de8ccc8e1
Change scope of scaleUp metric
2017-07-18 12:17:51 +02:00
aa1771107e
change scope of findUnneeded metric
2017-07-07 16:30:59 +02:00
7697d5345a
cluster-autoscaler: Fix scale-down when the node group auto-discovery feature is enabled
...
By fixing CA not to reset `StaticAutoscaler` state before each iteration so that it remembers last scale-up/down time which is used to throttle scale-down, which is causing the issue.
2017-06-22 10:25:37 +09:00
2cd532ebfe
Don't calculate utilization and run scale down simulations for unmanaged nodes
2017-06-20 16:57:30 +02:00
fe514ed75d
Make status configmap respect namespace parameter
2017-06-14 14:07:13 +02:00
69c77791a2
Fix error types
2017-06-12 21:26:50 +02:00
e2e171b7b7
Enable pricing in expander factory
2017-06-09 11:09:43 -07:00
58cdfa1702
Updated log levels in main loop
2017-05-18 14:09:15 +02:00
3f8ca51768
Use typed errors in scale down
2017-05-18 14:09:15 +02:00
7f5c7ed3a2
Used typed errors in scale up code
...
Updated some of the functions called by scale up
to return new errors as required.
2017-05-18 14:09:15 +02:00
f716a7e496
Add typed errors; add errors_total metric
...
To keep reasonable commit size only top-level files use
new errors. Will add them in other files in next commits.
2017-05-18 14:09:15 +02:00
d9bf5aacd7
Use TemplateNodeInfo in scale up
2017-05-16 11:45:05 +02:00
4cdf06ea94
Added CA metrics related to autoscaler execution
2017-05-11 14:51:04 +02:00
83ef3d2be3
Added CA metrics related to cluster state
2017-05-11 13:54:04 +02:00
5304e9af21
cluster-autoscaler: Fix typos in comments
2017-05-10 11:22:15 +09:00
Łukasz Osipiuk
dasydong
Kubernetes Prow Robot
Vivek Bagade
Jarvis-Zhou
Thomas Hartland
Julien Balestra
Krzysztof Jastrzebski
Łukasz Piątkowski
Jakub Tużnik
t-qini
Pengfei Ni
silenceper
Chris Bradfield
Jiaxin Shan
Aleksandra Malinowska
Uday Ruddarraju
Jacek Kaniuk
Maciej Pytel
Steve Scaffidi
Beata Skiba
Karol Gołąb
anniedy
Marcin Wielgus
Edward Tsang
Matt Terry
Ivan Towlson
Yusuke Kuoka