kops/docs/releases/1.24-NOTES.md

## Release notes for kOps 1.24 series

# Significant changes

## Hetzner Cloud support

Adds experimental Hetzner Cloud support. To get started check the [docs](../getting_started/hetzner.md)

## Karpenter support

By enabling the `Karpenter` feature flag, users can now create InstanceGroups managed by [Karpenter](https://karpenter.sh):

```
spec:
  manager: Karpenter
```

You can also start a Karpenter-only cluster with `kops create cluster --instance-manager=karpenter ...`

kOps will directly manage the Karpenter Provisioner resources. Read more about how Karpenter works on kOps in the [Karpenter docs](https://kops.sigs.k8s.io/operations/karpenter/).

## Other significant changes

* Adds support for Ubuntu 22.04

* Adds support for Rocky Linux 8

* Adds support for overriding the Kubernetes version when upgrading a cluster by using the `--kubernetes-version` flag.

* The minimum version for the Terraform AWS Provider has been bumped to 4.0.0 to address the deprecation of the aws_s3_bucket_object resource and its replacement with the aws_s3_object resource. Such resources will be destroyed and recreated without downtime when applying the changes. 

* ARM64 support for nvidia device driver. Nvidia nodes on ARM64 requires Ubuntu 22.04 AMIs.

# Breaking changes

* The nfs-common/nfs-utils package is no longer installed by default. Use the [packages](https://kops.sigs.k8s.io/instance_groups/#packages) option at instance group level to add it back. 

## Control plane taints and labels

As of Kubernetes version 1.24, the control plane (formerly master) nodes no longer have the deprecated `node-role.kubernetes.io/master` label.
The deprecated `node-role.kubernetes.io/master` taint has been replaced by `node-role.kubernetes.io/control-plane`. If you run your own workload on the control plane, you have to adjust your Pod spec to accommodate for this change.

The following shows a node affinity/node label selector and tolerations that works with both new and old control plane nodes:

```yaml
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: node-role.kubernetes.io/control-plane
            operator: Exists
        - matchExpressions:
          - key: node-role.kubernetes.io/master
            operator: Exists
  tolerations:
  - key: node-role.kubernetes.io/control-plane
    operator: Exists
  - key: node-role.kubernetes.io/master
    operator: Exists
```

## Removing the `kubernetes.io/role` label

The deprecated `kubernetes.io/role` label has been removed for all roles as of Kubernetes version 1.24. Use `node-role.kubernetes.io/<role>` label instead.

## Cert Manager removes old API versions

Cert Manager upgraded from 1.6 to 1.8. This has backwards-breaking changes. See upgrading from [1.6 to 1.7](https://cert-manager.io/docs/installation/upgrading/upgrading-1.6-1.7/) and [1.[1.7 to 1.8](https://cert-manager.io/docs/installation/upgrading/upgrading-1.7-1.8/).

In particular, if you are using the snapshot-controller addon, upgrade your cluster to kOps 1.23 before upgrading to kOps 1.24 to ensure the certificate has the correct API version.

## Other breaking changes

* Support for Kubernetes version 1.18 has been removed.

* Support for Aliyun/Alibaba Cloud has been removed.

* Support for Docker has been removed for Kubernetes 1.24+. See https://kubernetes.io/blog/2020/12/02/dockershim-faq

* Cluster Autoscaler can no longer run on spot instances. This is to avoid cluster autoscaler not being scheduled if the spot instances terminate.

# Required actions

# Deprecations

* Support for Kubernetes version 1.19 is deprecated and will be removed in kOps 1.25.

* Support for Kubernetes version 1.20 is deprecated and will be removed in kOps 1.26.

* All legacy addons are deprecated in favor of managed addons, including the [metrics server addon](https://github.com/kubernetes/kops/tree/master/addons/metrics-server) and the [autoscaler addon](https://github.com/kubernetes/kops/tree/master/addons/cluster-autoscaler).

* Due to lack of maintainers, the CloudFormation support has been deprecated. The current implementation will be left as-is until the implementation needs updates or otherwise becomes incompatible. At that point, it will be removed. We very much welcome anyone willing to contribute to this target.

* Support for Docker has been removed for Kubernetes 1.24+. See https://kubernetes.io/blog/2020/12/02/dockershim-faq

* The minimum supported version of the AWS EBS CSI Driver is now v1.3.1.

# Other changes of note

# Full change list since 1.23.0 release

## 1.24.0-alpha.1 to 1.24.0-alpha.2

* Update release notes and minimum k8s version [@hakman](https://github.com/hakman) [#12929](https://github.com/kubernetes/kops/pull/12929)
* kops auth-plugin: need to clear any existing password / key [@justinsb](https://github.com/justinsb) [#12921](https://github.com/kubernetes/kops/pull/12921)
* Add integration test for k8s 1.24 [@olemarkus](https://github.com/olemarkus) [#12930](https://github.com/kubernetes/kops/pull/12930)
* Only shellcheck files [@olemarkus](https://github.com/olemarkus) [#12931](https://github.com/kubernetes/kops/pull/12931)
* Do not set insecure-port as of k8s 1.20 [@olemarkus](https://github.com/olemarkus) [#12926](https://github.com/kubernetes/kops/pull/12926)
* tests: Improve logging on test failure [@justinsb](https://github.com/justinsb) [#12933](https://github.com/kubernetes/kops/pull/12933)
* nodeup: store the CloudProvider in the context [@justinsb](https://github.com/justinsb) [#12923](https://github.com/kubernetes/kops/pull/12923)
* bazel: always build with pure (CGO_ENABLED=0) [@justinsb](https://github.com/justinsb) [#12934](https://github.com/kubernetes/kops/pull/12934)
* nodeup: print more info on hash mismatches [@justinsb](https://github.com/justinsb) [#12935](https://github.com/kubernetes/kops/pull/12935)
* PKI library: Add initial support for EC keys [@justinsb](https://github.com/justinsb) [#12936](https://github.com/kubernetes/kops/pull/12936)
* Recognize debian bullseye as having "broken" resolv.conf [@justinsb](https://github.com/justinsb) [#12937](https://github.com/kubernetes/kops/pull/12937)
* Remove code for now-unsupported Kubernetes 1.18 [@johngmyers](https://github.com/johngmyers) [#12939](https://github.com/kubernetes/kops/pull/12939)
* Add missing k8s 1.18 relnote [@johngmyers](https://github.com/johngmyers) [#12938](https://github.com/kubernetes/kops/pull/12938)
* Remove obsolete, redundant secrets.md [@johngmyers](https://github.com/johngmyers) [#12942](https://github.com/kubernetes/kops/pull/12942)
* Drop support for Weave as of k8s 1.23 [@johngmyers](https://github.com/johngmyers) [#12941](https://github.com/kubernetes/kops/pull/12941)
* Remove support for Aliyun/Alibaba Cloud [@johngmyers](https://github.com/johngmyers) [#12944](https://github.com/kubernetes/kops/pull/12944)
* Document CoreDNS configuration settings [@recollir](https://github.com/recollir) [#12914](https://github.com/kubernetes/kops/pull/12914)
* Update name of kubernetes-ca keypair in documentation [@johngmyers](https://github.com/johngmyers) [#12943](https://github.com/kubernetes/kops/pull/12943)
* Revert "Recognize debian bullseye as having "broken" resolv.conf" [@olemarkus](https://github.com/olemarkus) [#12947](https://github.com/kubernetes/kops/pull/12947)
* Set the default LT version to the new LT version [@olemarkus](https://github.com/olemarkus) [#12932](https://github.com/kubernetes/kops/pull/12932)
* Make service topology for cilium configurable [@olemarkus](https://github.com/olemarkus) [#12918](https://github.com/kubernetes/kops/pull/12918)
* gce: ServiceAccount task [@justinsb](https://github.com/justinsb) [#12950](https://github.com/kubernetes/kops/pull/12950)
* Update Calico and Canal to v3.21.2 [@hakman](https://github.com/hakman) [#12951](https://github.com/kubernetes/kops/pull/12951)
* Update Go to v1.17.5 [@hakman](https://github.com/hakman) [#12954](https://github.com/kubernetes/kops/pull/12954)
* Skip IPv6 LB test in the k/s e2e [@hakman](https://github.com/hakman) [#12953](https://github.com/kubernetes/kops/pull/12953)
* GCE: Task for StorageBucket IAM [@justinsb](https://github.com/justinsb) [#12958](https://github.com/kubernetes/kops/pull/12958)
* GCE: Project IAM Binding task [@justinsb](https://github.com/justinsb) [#12959](https://github.com/kubernetes/kops/pull/12959)
* add verify-golangci-lint.sh script [@rlankfo](https://github.com/rlankfo) [#12892](https://github.com/kubernetes/kops/pull/12892)
* Hubble relay should not tolerate anything [@olemarkus](https://github.com/olemarkus) [#12963](https://github.com/kubernetes/kops/pull/12963)
* Do not explicitly skip Dashboard tests [@hakman](https://github.com/hakman) [#12962](https://github.com/kubernetes/kops/pull/12962)
* Do not skip NodePort tests for Calico [@hakman](https://github.com/hakman) [#12960](https://github.com/kubernetes/kops/pull/12960)
* Remove verify-staticcheck [@rifelpet](https://github.com/rifelpet) [#12965](https://github.com/kubernetes/kops/pull/12965)
* wait for instances to drain from classic LB [@heybronson](https://github.com/heybronson) [#12902](https://github.com/kubernetes/kops/pull/12902)
* Support Karpenter [@olemarkus](https://github.com/olemarkus) [#12906](https://github.com/kubernetes/kops/pull/12906)
* Update containerd to v1.6.0-beta.4 [@hakman](https://github.com/hakman) [#12968](https://github.com/kubernetes/kops/pull/12968)
* Update controller-runtime to v0.11.0 [@hakman](https://github.com/hakman) [#12967](https://github.com/kubernetes/kops/pull/12967)
* Add missing permissions [@olemarkus](https://github.com/olemarkus) [#12977](https://github.com/kubernetes/kops/pull/12977)
* Do not skip HPA tests [@hakman](https://github.com/hakman) [#12972](https://github.com/kubernetes/kops/pull/12972)
* Do not skip RuntimeClass tests [@hakman](https://github.com/hakman) [#12974](https://github.com/kubernetes/kops/pull/12974)
* gce: Use ServiceAccount task when building model [@justinsb](https://github.com/justinsb) [#12978](https://github.com/kubernetes/kops/pull/12978)
* Quote values and remove limits in karpenter provisioners [@olemarkus](https://github.com/olemarkus) [#12979](https://github.com/kubernetes/kops/pull/12979)
* Promote alpha with December releases [@olemarkus](https://github.com/olemarkus) [#12984](https://github.com/kubernetes/kops/pull/12984)
* gce: map multiple serviceaccounts [@justinsb](https://github.com/justinsb),[@hakman](https://github.com/hakman) [#12982](https://github.com/kubernetes/kops/pull/12982)
* Defend against nil containerd [@justinsb](https://github.com/justinsb) [#12990](https://github.com/kubernetes/kops/pull/12990)
* Remove unused TemplateResource interface [@justinsb](https://github.com/justinsb) [#12989](https://github.com/kubernetes/kops/pull/12989)
* Avoid double-encoding templates [@justinsb](https://github.com/justinsb) [#12991](https://github.com/kubernetes/kops/pull/12991)
* Refactor nodeup script to avoid action-at-a-distance [@justinsb](https://github.com/justinsb) [#12993](https://github.com/kubernetes/kops/pull/12993)
* gce: use per InstanceGroup serviceaccounts [@justinsb](https://github.com/justinsb) [#12988](https://github.com/kubernetes/kops/pull/12988)
* dep: update github.com/pkg/sftp [@justinsb](https://github.com/justinsb) [#12996](https://github.com/kubernetes/kops/pull/12996)
* Create helper functions for parsing public keys [@justinsb](https://github.com/justinsb) [#12999](https://github.com/kubernetes/kops/pull/12999)
* Use terraform literals in GCP service account references [@rifelpet](https://github.com/rifelpet) [#12995](https://github.com/kubernetes/kops/pull/12995)
* kops-controller: use controller-runtime manager [@justinsb](https://github.com/justinsb) [#12997](https://github.com/kubernetes/kops/pull/12997)
* gce: clean up networking objects by reference [@justinsb](https://github.com/justinsb) [#12987](https://github.com/kubernetes/kops/pull/12987)
* componentconfig: expose advertise-address flag for kube-apiserver [@justinsb](https://github.com/justinsb) [#12998](https://github.com/kubernetes/kops/pull/12998)
* Do not allow docker on k8s 1.24+ [@olemarkus](https://github.com/olemarkus) [#12927](https://github.com/kubernetes/kops/pull/12927)
* Ignore images hosted in private ECR repositories as containerd cannot pull these [@olemarkus](https://github.com/olemarkus) [#13000](https://github.com/kubernetes/kops/pull/13000)
* Skip RuntimeClass tests for older Kubernetes versions [@hakman](https://github.com/hakman) [#13003](https://github.com/kubernetes/kops/pull/13003)
* Various nill pointer fixes for karpenter [@olemarkus](https://github.com/olemarkus) [#12973](https://github.com/kubernetes/kops/pull/12973)
* Set Resource Based Naming on managed subnets [@johngmyers](https://github.com/johngmyers) [#12864](https://github.com/kubernetes/kops/pull/12864)
* Add kubetest2-kops flags for overriding instance group fields [@rifelpet](https://github.com/rifelpet) [#13005](https://github.com/kubernetes/kops/pull/13005)
* Support creating dualstack internal NLBs [@johngmyers](https://github.com/johngmyers) [#13006](https://github.com/kubernetes/kops/pull/13006)
* Skip SCTP check for all versions of k8s 1.23/1.24 [@olemarkus](https://github.com/olemarkus) [#13008](https://github.com/kubernetes/kops/pull/13008)
* Use spread constraints rather than affinity to spread pods [@olemarkus](https://github.com/olemarkus) [#12961](https://github.com/kubernetes/kops/pull/12961)
* Bump karpenter to 0.5.3 and RBN support [@olemarkus](https://github.com/olemarkus) [#13002](https://github.com/kubernetes/kops/pull/13002)
* Validate IGs more strictly after defaults have applied [@olemarkus](https://github.com/olemarkus) [#12660](https://github.com/kubernetes/kops/pull/12660)
* Karpenter template fix [@olemarkus](https://github.com/olemarkus) [#13009](https://github.com/kubernetes/kops/pull/13009)
* staticcheck cleanup: fixup nodeup/pkg/model [@justinsb](https://github.com/justinsb) [#13013](https://github.com/kubernetes/kops/pull/13013)
* nodeup bash script: use explicit return code [@justinsb](https://github.com/justinsb) [#13012](https://github.com/kubernetes/kops/pull/13012)
* Prevent creation of unsupported etcd clusters [@olemarkus](https://github.com/olemarkus) [#13011](https://github.com/kubernetes/kops/pull/13011)
* Create cgroups for kube and runtime if configured [@olemarkus](https://github.com/olemarkus) [#12917](https://github.com/kubernetes/kops/pull/12917)
* Do not install ClusterRole and binding used by in-tree volume provider if CSI is used [@olemarkus](https://github.com/olemarkus) [#13010](https://github.com/kubernetes/kops/pull/13010)
* kubetest2 - Use the same binary path and env when fetching IGs [@rifelpet](https://github.com/rifelpet) [#13018](https://github.com/kubernetes/kops/pull/13018)
* Use fi.Keyset instead of passing tasks around [@justinsb](https://github.com/justinsb) [#12992](https://github.com/kubernetes/kops/pull/12992)
* add instance connection draining for NLBs [@heybronson](https://github.com/heybronson) [#12966](https://github.com/kubernetes/kops/pull/12966)
* Use kubelet --non-masquerade-cidr only for Docker with kubenet [@hakman](https://github.com/hakman) [#13007](https://github.com/kubernetes/kops/pull/13007)
* Fix dangling ENIs from AWS VPC CNI  [@olemarkus](https://github.com/olemarkus) [#13021](https://github.com/kubernetes/kops/pull/13021)
* Update k8s dependencies to v1.23.1 [@hakman](https://github.com/hakman) [#13022](https://github.com/kubernetes/kops/pull/13022)
* Improve HA for various addons [@olemarkus](https://github.com/olemarkus) [#13027](https://github.com/kubernetes/kops/pull/13027)
* Add a CLI flag for creating one karpenter-managed IG for worker nodes instead of ASG-managed ones [@olemarkus](https://github.com/olemarkus) [#12975](https://github.com/kubernetes/kops/pull/12975)
* Allow IPv6-only subnets [@johngmyers](https://github.com/johngmyers) [#13026](https://github.com/kubernetes/kops/pull/13026)
* Support specifying instance requirements per IG [@olemarkus](https://github.com/olemarkus) [#13019](https://github.com/kubernetes/kops/pull/13019)
* Remove TerraformJSON feature flag [@rifelpet](https://github.com/rifelpet) [#13029](https://github.com/kubernetes/kops/pull/13029)
* LBC has to run on the control plane, so set replicas accordingly [@olemarkus](https://github.com/olemarkus) [#13033](https://github.com/kubernetes/kops/pull/13033)
* Fix various typos related to karpenter [@olemarkus](https://github.com/olemarkus) [#13035](https://github.com/kubernetes/kops/pull/13035)
* Kube components log to stdout [@olemarkus](https://github.com/olemarkus) [#13038](https://github.com/kubernetes/kops/pull/13038)
* Identify pending instances [@olemarkus](https://github.com/olemarkus) [#13040](https://github.com/kubernetes/kops/pull/13040)
* Add managed-by label to static kube-proxy pods [@olemarkus](https://github.com/olemarkus) [#13039](https://github.com/kubernetes/kops/pull/13039)
* Prefix karpenter logging-config name [@olemarkus](https://github.com/olemarkus) [#13037](https://github.com/kubernetes/kops/pull/13037)
* gce: don't set per-IG permissions when using shared account [@justinsb](https://github.com/justinsb) [#13043](https://github.com/kubernetes/kops/pull/13043)
* Add documentation on karpenter [@olemarkus](https://github.com/olemarkus) [#13036](https://github.com/kubernetes/kops/pull/13036)
* external CCM for GCE [@jiahuif](https://github.com/jiahuif) [#13017](https://github.com/kubernetes/kops/pull/13017)
* Migrate to GCE CCM in k8s 1.24 [@johngmyers](https://github.com/johngmyers) [#13045](https://github.com/kubernetes/kops/pull/13045)
* Fix OpenStack SecurityGroupRule/LB When CIDR is IPv6 [@iGene](https://github.com/iGene) [#13032](https://github.com/kubernetes/kops/pull/13032)
* update deps [@zetaab](https://github.com/zetaab) [#13047](https://github.com/kubernetes/kops/pull/13047)
* Bump Cluster Autoscaler and update manifest [@olemarkus](https://github.com/olemarkus) [#13050](https://github.com/kubernetes/kops/pull/13050)
* Use instance requirements with Karpenter [@olemarkus](https://github.com/olemarkus) [#13031](https://github.com/kubernetes/kops/pull/13031)
* force update dependencies [@zetaab](https://github.com/zetaab) [#13055](https://github.com/kubernetes/kops/pull/13055)
* Enhance AddHostPathMapping to support a fluent style [@justinsb](https://github.com/justinsb) [#13062](https://github.com/kubernetes/kops/pull/13062)
* addons: support for kopeio-networking addon [@justinsb](https://github.com/justinsb) [#12727](https://github.com/kubernetes/kops/pull/12727)
* Use latest GCP CCM for k8s 1.24 [@johngmyers](https://github.com/johngmyers) [#13066](https://github.com/kubernetes/kops/pull/13066)
* Add action for automatically tagging releases [@johngmyers](https://github.com/johngmyers) [#12805](https://github.com/kubernetes/kops/pull/12805)
* Bump external-snapshotted to v5.0.0 [@olemarkus](https://github.com/olemarkus) [#13067](https://github.com/kubernetes/kops/pull/13067)
* Release 1.24.0-alpha.2 [@johngmyers](https://github.com/johngmyers) [#13069](https://github.com/kubernetes/kops/pull/13069)

## 1.24.0-alpha.2 to 1.24.0-alpha.3

* Release notes for 1.24.0-alpha.2 [@johngmyers](https://github.com/johngmyers) [#13070](https://github.com/kubernetes/kops/pull/13070)
* Update release process for automatic tagging [@johngmyers](https://github.com/johngmyers) [#13075](https://github.com/kubernetes/kops/pull/13075)
* Remove temporary restrictions on automatically tagging releases [@johngmyers](https://github.com/johngmyers) [#13071](https://github.com/kubernetes/kops/pull/13071)
* add flatcar note related to additionalUserData [@shubhindia](https://github.com/shubhindia) [#13061](https://github.com/kubernetes/kops/pull/13061)
* Drain OpenStack loadbalancers [@zetaab](https://github.com/zetaab) [#12983](https://github.com/kubernetes/kops/pull/12983)
* Extend terraform support for IPv6 [@rifelpet](https://github.com/rifelpet) [#13028](https://github.com/kubernetes/kops/pull/13028)
* Update containerd to v1.6.0-beta.5 [@hakman](https://github.com/hakman) [#13084](https://github.com/kubernetes/kops/pull/13084)
* Release notes for 1.22.3 [@johngmyers](https://github.com/johngmyers) [#13085](https://github.com/kubernetes/kops/pull/13085)
* Spotinst: Update `spotinst/ocean-controller` to v1.0.81 [@liranp](https://github.com/liranp) [#13086](https://github.com/kubernetes/kops/pull/13086)
* Support price and priority cluster-autoscaler expanders [@danports](https://github.com/danports) [#13081](https://github.com/kubernetes/kops/pull/13081)
* Update containerd to v1.6.0-rc.0  [@hakman](https://github.com/hakman) [#13098](https://github.com/kubernetes/kops/pull/13098)
* decrease the openstack monitoring default timeout [@zetaab](https://github.com/zetaab) [#13097](https://github.com/kubernetes/kops/pull/13097)
* Don't try to add node name to instances without node object [@olemarkus](https://github.com/olemarkus) [#13106](https://github.com/kubernetes/kops/pull/13106)
* fix ipv4+ipv6 sec groups/listeners in OpenStack [@zetaab](https://github.com/zetaab) [#13093](https://github.com/kubernetes/kops/pull/13093)
* Do not create an IAM role for dns-controller on gossip clusters [@olemarkus](https://github.com/olemarkus) [#13110](https://github.com/kubernetes/kops/pull/13110)
* Add ipv6 to relnotes [@olemarkus](https://github.com/olemarkus) [#13088](https://github.com/kubernetes/kops/pull/13088)
* Use IPv6-only subnets for worker nodes in private IPv6 topology [@johngmyers](https://github.com/johngmyers) [#13030](https://github.com/kubernetes/kops/pull/13030)
* Remove networking flags as of k8s 1.24 [@olemarkus](https://github.com/olemarkus) [#13120](https://github.com/kubernetes/kops/pull/13120)
* Create helper function for ec2 create/tag-on-create IAM permissions [@olemarkus](https://github.com/olemarkus) [#13104](https://github.com/kubernetes/kops/pull/13104)
* Add DescribeRegions to nodeup privs [@olemarkus](https://github.com/olemarkus) [#13114](https://github.com/kubernetes/kops/pull/13114)
* Remove featureflag for creating IPv6 clusters [@hakman](https://github.com/hakman) [#12788](https://github.com/kubernetes/kops/pull/12788)
* Preload channel versions from namespaces [@olemarkus](https://github.com/olemarkus) [#13049](https://github.com/kubernetes/kops/pull/13049)
* Don't set unsupported configs by default [@olemarkus](https://github.com/olemarkus) [#13111](https://github.com/kubernetes/kops/pull/13111)
* Update pause image to v3.6 [@hakman](https://github.com/hakman) [#13125](https://github.com/kubernetes/kops/pull/13125)
* Clean up kubelet networking flags for dockershim [@hakman](https://github.com/hakman) [#13128](https://github.com/kubernetes/kops/pull/13128)
* January bump of channels [@olemarkus](https://github.com/olemarkus) [#13130](https://github.com/kubernetes/kops/pull/13130)
* expose external ccm metrics for OpenStack [@zetaab](https://github.com/zetaab) [#13131](https://github.com/kubernetes/kops/pull/13131)
* Update to aws-sdk-go to v1.42.37 [@jinhong-](https://github.com/jinhong-) [#13132](https://github.com/kubernetes/kops/pull/13132)
* Fix recommended kops versions in channels [@olemarkus](https://github.com/olemarkus) [#13134](https://github.com/kubernetes/kops/pull/13134)
* Tag on create for remaining CCM privileges [@olemarkus](https://github.com/olemarkus) [#12911](https://github.com/kubernetes/kops/pull/12911)
* Bump metrics-server to 0.6.0 and enable HA mode [@olemarkus](https://github.com/olemarkus) [#13135](https://github.com/kubernetes/kops/pull/13135)
* OpenStack - Add loadbalancer pool monitor to API LB [@zetaab](https://github.com/zetaab) [#13096](https://github.com/kubernetes/kops/pull/13096)
* Bump CCM images [@olemarkus](https://github.com/olemarkus) [#13143](https://github.com/kubernetes/kops/pull/13143)
* Bump karpenter to 0.5.6 [@olemarkus](https://github.com/olemarkus) [#13151](https://github.com/kubernetes/kops/pull/13151)
* Promote alpha AMIs to stable [@yurrriq](https://github.com/yurrriq) [#13152](https://github.com/kubernetes/kops/pull/13152)
* Bump 1.23 version in alpha channel [@olemarkus](https://github.com/olemarkus) [#13153](https://github.com/kubernetes/kops/pull/13153)
* Add missing v prefix to default upgrade test version [@olemarkus](https://github.com/olemarkus) [#13155](https://github.com/kubernetes/kops/pull/13155)
* Bump cert-manager and related godep to 1.6.2 [@olemarkus](https://github.com/olemarkus) [#13154](https://github.com/kubernetes/kops/pull/13154)
* add node-drain-timeout flag to rolling-update [@heybronson](https://github.com/heybronson) [#13103](https://github.com/kubernetes/kops/pull/13103)
* Bump etcd-manager to v3.0.20220128 [@olemarkus](https://github.com/olemarkus) [#13158](https://github.com/kubernetes/kops/pull/13158)
* Replace deprecated aws.BackgroundContext with context.Background [@justinsb](https://github.com/justinsb) [#13162](https://github.com/kubernetes/kops/pull/13162)
* Fix nil pointer when IAM not populated [@justinsb](https://github.com/justinsb) [#13167](https://github.com/kubernetes/kops/pull/13167)
* JWKS / IRSA: Expose public ACLs to terraform [@justinsb](https://github.com/justinsb) [#13166](https://github.com/kubernetes/kops/pull/13166)
* [DigitalOcean] update ccm version to 0.1.36 [@srikiz](https://github.com/srikiz) [#13175](https://github.com/kubernetes/kops/pull/13175)
* Bump Ubuntu AMI in alpha [@olemarkus](https://github.com/olemarkus) [#13177](https://github.com/kubernetes/kops/pull/13177)
* Use etcd-manager pre-release until final release has been cut [@olemarkus](https://github.com/olemarkus) [#13183](https://github.com/kubernetes/kops/pull/13183)
* Bump karpenter to 0.6.0 [@olemarkus](https://github.com/olemarkus) [#13185](https://github.com/kubernetes/kops/pull/13185)
* More descriptive error message when public key file can't be opened [@nckturner](https://github.com/nckturner) [#13186](https://github.com/kubernetes/kops/pull/13186)
* update GCE default images [@zetaab](https://github.com/zetaab) [#13181](https://github.com/kubernetes/kops/pull/13181)
* Fix etcd-manager for ipv6 [@olemarkus](https://github.com/olemarkus) [#13191](https://github.com/kubernetes/kops/pull/13191)
* Update Calico and Canal to v3.21.4 [@hakman](https://github.com/hakman) [#13189](https://github.com/kubernetes/kops/pull/13189)
* Update to etcd-manager v3.0.20220203 [@justinsb](https://github.com/justinsb) [#13196](https://github.com/kubernetes/kops/pull/13196)
* Pull k8s-custom-iptables from k8s.gcr.io [@justinsb](https://github.com/justinsb) [#13194](https://github.com/kubernetes/kops/pull/13194)
* Add support for AB tests starting out with released kops version [@olemarkus](https://github.com/olemarkus) [#13174](https://github.com/kubernetes/kops/pull/13174)
* Update containerd to v1.6.0-rc.2 [@hakman](https://github.com/hakman) [#13198](https://github.com/kubernetes/kops/pull/13198)
* tests: ensure that we use ACLs with memfs [@justinsb](https://github.com/justinsb) [#13165](https://github.com/kubernetes/kops/pull/13165)
* Karpenter fixes [@olemarkus](https://github.com/olemarkus) [#13207](https://github.com/kubernetes/kops/pull/13207)
* Always enable Leader Election for cloud-controller-manager [@jiahuif](https://github.com/jiahuif) [#13187](https://github.com/kubernetes/kops/pull/13187)
* Use short commit sha for default stage location instead of git-describe [@olemarkus](https://github.com/olemarkus) [#13208](https://github.com/kubernetes/kops/pull/13208)
* use 1.23.1 ccm for openstack [@zetaab](https://github.com/zetaab) [#13136](https://github.com/kubernetes/kops/pull/13136)
* Document download of test versions [@olemarkus](https://github.com/olemarkus) [#13209](https://github.com/kubernetes/kops/pull/13209)
* Remove snapshot controller dependency on ebs csi driver [@olemarkus](https://github.com/olemarkus) [#13213](https://github.com/kubernetes/kops/pull/13213)
* fix KCM LogLevel setting not honored [@jiahuif](https://github.com/jiahuif) [#13218](https://github.com/kubernetes/kops/pull/13218)
* Fix CSI migration feature gates [@olemarkus](https://github.com/olemarkus) [#13203](https://github.com/kubernetes/kops/pull/13203)
* CCM: use flagbuilder instead of manually building argv [@jiahuif](https://github.com/jiahuif) [#13219](https://github.com/kubernetes/kops/pull/13219)
* Update containerd to v1.6.0-rc.3 [@hakman](https://github.com/hakman) [#13224](https://github.com/kubernetes/kops/pull/13224)
* Promote alpha to stable [@MoShitrit](https://github.com/MoShitrit) [#13227](https://github.com/kubernetes/kops/pull/13227)
* always enable Leader Election for openstack CCM [@jiahuif](https://github.com/jiahuif) [#13220](https://github.com/kubernetes/kops/pull/13220)
* Update aws node termination handler to 1.14.0 [@ryan-dyer-sp](https://github.com/ryan-dyer-sp),[@ryan-dyer](https://github.com/ryan-dyer) [#13092](https://github.com/kubernetes/kops/pull/13092)
* [Issue-12293] Fix json output to keep it consistent for single or multiple objects [@srikiz](https://github.com/srikiz) [#13188](https://github.com/kubernetes/kops/pull/13188)
* Fix irsa for k8s < 1.20 [@olemarkus](https://github.com/olemarkus) [#13212](https://github.com/kubernetes/kops/pull/13212)
* enable pruning for CCM [@jiahuif](https://github.com/jiahuif) [#13235](https://github.com/kubernetes/kops/pull/13235)
* Add support for graceful node shutdown [@olemarkus](https://github.com/olemarkus) [#12994](https://github.com/kubernetes/kops/pull/12994)
* allow specify GCP project via env. [@jiahuif](https://github.com/jiahuif) [#13237](https://github.com/kubernetes/kops/pull/13237)
* KCM should not run with leader migraton when aws ccm is enabled [@olemarkus](https://github.com/olemarkus) [#13241](https://github.com/kubernetes/kops/pull/13241)
* Do not enable graceful shutdown if k8s version < 1.21 [@olemarkus](https://github.com/olemarkus) [#13242](https://github.com/kubernetes/kops/pull/13242)
* Update metrics-server e2e test for 0.6.0 [@olemarkus](https://github.com/olemarkus) [#13243](https://github.com/kubernetes/kops/pull/13243)
* Install runc from opencontainers/runc [@hakman](https://github.com/hakman) [#13240](https://github.com/kubernetes/kops/pull/13240)
* Fix nilpointer when graceful shutdown is not configured [@olemarkus](https://github.com/olemarkus) [#13246](https://github.com/kubernetes/kops/pull/13246)
* Install contained from the release package  [@hakman](https://github.com/hakman) [#13248](https://github.com/kubernetes/kops/pull/13248)
* CCM: allow setting Controllers for cloudControllerManagerConfig [@jiahuif](https://github.com/jiahuif) [#13252](https://github.com/kubernetes/kops/pull/13252)
* CCM: add livenessProbe for GCP CCM [@jiahuif](https://github.com/jiahuif) [#13253](https://github.com/kubernetes/kops/pull/13253)
* E2E HA Upgrade/Rollback for Leader Migration [@jiahuif](https://github.com/jiahuif) [#13251](https://github.com/kubernetes/kops/pull/13251)
* Bump AWS CNI to 1.10.2 [@MoShitrit](https://github.com/MoShitrit) [#13228](https://github.com/kubernetes/kops/pull/13228)
* Update supported distros for IPv6 [@hakman](https://github.com/hakman) [#13256](https://github.com/kubernetes/kops/pull/13256)
* Karpenter on kOps will now use approperiate max pods [@olemarkus](https://github.com/olemarkus) [#13178](https://github.com/kubernetes/kops/pull/13178)
* Allow PrefixList for sshAccess and kubernetesApiAccess [@hierynomus](https://github.com/hierynomus) [#13113](https://github.com/kubernetes/kops/pull/13113)
* service account workaround for gce [@jiahuif](https://github.com/jiahuif) [#13261](https://github.com/kubernetes/kops/pull/13261)
* GCP API health checks [@zetaab](https://github.com/zetaab) [#13199](https://github.com/kubernetes/kops/pull/13199)
* Update containerd to v1.6.0 [@hakman](https://github.com/hakman) [#13262](https://github.com/kubernetes/kops/pull/13262)
* re-organize Leader Migration test with exec tester [@jiahuif](https://github.com/jiahuif) [#13265](https://github.com/kubernetes/kops/pull/13265)
* Update LBC to 2.4.0 [@olemarkus](https://github.com/olemarkus) [#13267](https://github.com/kubernetes/kops/pull/13267)
* Enable RBN with AWS CCM 1.22.0-alpha.1 [@johngmyers](https://github.com/johngmyers) [#13268](https://github.com/kubernetes/kops/pull/13268)
* Disable some flags in kube-apiserver when logging-format is not text [@h3poteto](https://github.com/h3poteto) [#13264](https://github.com/kubernetes/kops/pull/13264)
* kops: Leader Migration testing: run with pure kubetest2 [@jiahuif](https://github.com/jiahuif) [#13276](https://github.com/kubernetes/kops/pull/13276)
* Bump k8s versions in alpha with Feb 2022 releases [@MoShitrit](https://github.com/MoShitrit) [#13275](https://github.com/kubernetes/kops/pull/13275)
* Validate taints in IG spec [@olemarkus](https://github.com/olemarkus) [#13266](https://github.com/kubernetes/kops/pull/13266)
* test: use `T.TempDir` to create temporary test directory [@Juneezee](https://github.com/Juneezee) [#13283](https://github.com/kubernetes/kops/pull/13283)
* Do not create a cert-manager namespace [@olemarkus](https://github.com/olemarkus) [#13284](https://github.com/kubernetes/kops/pull/13284)
* Add missing permissions to aws lbc for irsa [@olemarkus](https://github.com/olemarkus) [#13280](https://github.com/kubernetes/kops/pull/13280)
* [DigitalOcean] Implement new VPC if network-cidr flag is specified [@srikiz](https://github.com/srikiz) [#13060](https://github.com/kubernetes/kops/pull/13060)
* Use current tree in presubmit upgrade jobs if version B is latest [@olemarkus](https://github.com/olemarkus) [#13290](https://github.com/kubernetes/kops/pull/13290)
* Release notes for 1.22.4 [@justinsb](https://github.com/justinsb) [#13294](https://github.com/kubernetes/kops/pull/13294)
* alpha channel: recommend kOps 1.22.4 [@justinsb](https://github.com/justinsb) [#13296](https://github.com/kubernetes/kops/pull/13296)
* docs: add hubble ui helm chart deployment [@eddycharly](https://github.com/eddycharly) [#13299](https://github.com/kubernetes/kops/pull/13299)
* cleanup GCP Cluster Service Accounts [@zetaab](https://github.com/zetaab) [#13201](https://github.com/kubernetes/kops/pull/13201)
* docs for release process shouldn't assume remotes [@justinsb](https://github.com/justinsb) [#13295](https://github.com/kubernetes/kops/pull/13295)
* Release notes for 1.23.0-beta.2 [@hakman](https://github.com/hakman) [#13303](https://github.com/kubernetes/kops/pull/13303)
* Add support to install EKS Pod Identity Webhook [@h3poteto](https://github.com/h3poteto),[@olemarkus](https://github.com/olemarkus) [#13176](https://github.com/kubernetes/kops/pull/13176)
* Update kubetest2 deps [@olemarkus](https://github.com/olemarkus) [#13314](https://github.com/kubernetes/kops/pull/13314)
* use own function to define CSI image version [@zetaab](https://github.com/zetaab) [#13311](https://github.com/kubernetes/kops/pull/13311)
* Add support for ed25519 keys in AWS [@aclevername](https://github.com/aclevername) [#13304](https://github.com/kubernetes/kops/pull/13304)
* Bump AWS SDK to v1.43.11 [@olemarkus](https://github.com/olemarkus) [#13322](https://github.com/kubernetes/kops/pull/13322)
* Make cloudProvider a struct in v1alpha3 API [@johngmyers](https://github.com/johngmyers) [#13059](https://github.com/kubernetes/kops/pull/13059)
* Update containerd to v1.6.1 [@hakman](https://github.com/hakman) [#13325](https://github.com/kubernetes/kops/pull/13325)
* Fix GCE service account creation [@zetaab](https://github.com/zetaab) [#13310](https://github.com/kubernetes/kops/pull/13310)
* Use proper image and add health check [@olemarkus](https://github.com/olemarkus) [#13328](https://github.com/kubernetes/kops/pull/13328)
* Update stable and alpha channels [@olemarkus](https://github.com/olemarkus) [#13334](https://github.com/kubernetes/kops/pull/13334)
* Release notes for 1.21.5 [@hakman](https://github.com/hakman) [#13336](https://github.com/kubernetes/kops/pull/13336)
* Add e2e for pod identity webhook [@olemarkus](https://github.com/olemarkus) [#13335](https://github.com/kubernetes/kops/pull/13335)
* Add webhook notes + some docs changes [@olemarkus](https://github.com/olemarkus) [#13338](https://github.com/kubernetes/kops/pull/13338)
* Only delete node object on GCE [@olemarkus](https://github.com/olemarkus) [#13289](https://github.com/kubernetes/kops/pull/13289)
* Release notes for 1.23.0 [@hakman](https://github.com/hakman) [#13340](https://github.com/kubernetes/kops/pull/13340)
* Bump AWS CCM to 1.22.0-alpha.2 [@olemarkus](https://github.com/olemarkus) [#13342](https://github.com/kubernetes/kops/pull/13342)
* Bump CCM 1.22 image. Use the 1.23 image for 1.24 due to latest being broken [@olemarkus](https://github.com/olemarkus) [#13357](https://github.com/kubernetes/kops/pull/13357)
* Update channels [@hakman](https://github.com/hakman) [#13356](https://github.com/kubernetes/kops/pull/13356)
* Recommend enabling IRSA for new clusters [@olemarkus](https://github.com/olemarkus) [#12976](https://github.com/kubernetes/kops/pull/12976)
* Post 1.23.0 release doc updates [@johngmyers](https://github.com/johngmyers) [#13359](https://github.com/kubernetes/kops/pull/13359)
* Add user to container securityContext and remove command [@olemarkus](https://github.com/olemarkus) [#13343](https://github.com/kubernetes/kops/pull/13343)
* [Digital Ocean] e2e tests - Fix seeding for generating random zones [@srikiz](https://github.com/srikiz) [#13362](https://github.com/kubernetes/kops/pull/13362)
* wait for all targetGroups to drain [@heybronson](https://github.com/heybronson) [#13363](https://github.com/kubernetes/kops/pull/13363)
* Support GPU in OpenStack [@zetaab](https://github.com/zetaab) [#13330](https://github.com/kubernetes/kops/pull/13330)
* Add missing permissions to aws lbc for IP targeting [@olemarkus](https://github.com/olemarkus) [#13369](https://github.com/kubernetes/kops/pull/13369)
* If kubetest2 fails cluster validation, we run down before exiting [@olemarkus](https://github.com/olemarkus) [#13373](https://github.com/kubernetes/kops/pull/13373)
* If image is empty, have kops upgrade fill it in [@olemarkus](https://github.com/olemarkus) [#13374](https://github.com/kubernetes/kops/pull/13374)
* Update channels [@hakman](https://github.com/hakman) [#13379](https://github.com/kubernetes/kops/pull/13379)
* Update HPA docs [@ddelange](https://github.com/ddelange) [#13367](https://github.com/kubernetes/kops/pull/13367)
* Clean up nodeup targets [@olemarkus](https://github.com/olemarkus) [#13370](https://github.com/kubernetes/kops/pull/13370)
* Upgrade aws-iam-authenticator to v0.5.5 [@glebiller](https://github.com/glebiller) [#13381](https://github.com/kubernetes/kops/pull/13381)
* Add protocol explicitly to services [@olemarkus](https://github.com/olemarkus) [#13383](https://github.com/kubernetes/kops/pull/13383)
* Allow duplicate taint keys [@olemarkus](https://github.com/olemarkus) [#13366](https://github.com/kubernetes/kops/pull/13366)
* [Digital Ocean] Remove sfo2 region from the list of supported DO regions [@srikiz](https://github.com/srikiz) [#13382](https://github.com/kubernetes/kops/pull/13382)
* Fix long role names [@olemarkus](https://github.com/olemarkus) [#13364](https://github.com/kubernetes/kops/pull/13364)
* Migrate to registry.k8s.io [@hakman](https://github.com/hakman) [#13380](https://github.com/kubernetes/kops/pull/13380)
* Remove oss-upload target since aliyun support has been removed [@olemarkus](https://github.com/olemarkus) [#13389](https://github.com/kubernetes/kops/pull/13389)
* dev: create scripts to make it easier to run e2e tests [@justinsb](https://github.com/justinsb) [#13161](https://github.com/kubernetes/kops/pull/13161)
* Remove pr target [@olemarkus](https://github.com/olemarkus) [#13392](https://github.com/kubernetes/kops/pull/13392)

## 1.24.0-alpha.3 to 1.24.0-alpha.4

* Release notes for 1.24.0-alpha.3 [@heybronson](https://github.com/heybronson) [#13375](https://github.com/kubernetes/kops/pull/13375)
* Spotinst: Ocean as a template for VNGs [@liranp](https://github.com/liranp) [#13234](https://github.com/kubernetes/kops/pull/13234)
* [Digital Ocean] Use available regions as per the latest DO documentation [@srikiz](https://github.com/srikiz) [#13394](https://github.com/kubernetes/kops/pull/13394)
* Build kops images with ko [@olemarkus](https://github.com/olemarkus) [#13341](https://github.com/kubernetes/kops/pull/13341)
* Clean up kops build targets [@olemarkus](https://github.com/olemarkus) [#13390](https://github.com/kubernetes/kops/pull/13390)
* update k8s dependencies [@heybronson](https://github.com/heybronson) [#13397](https://github.com/kubernetes/kops/pull/13397)
* Update golangci-lint to v1.45.0  [@hakman](https://github.com/hakman) [#13403](https://github.com/kubernetes/kops/pull/13403)
* Use k8s.gcr.io for k8s side-loaded images [@hakman](https://github.com/hakman) [#13402](https://github.com/kubernetes/kops/pull/13402)
* Use ko-builds for dev-upload [@olemarkus](https://github.com/olemarkus) [#13401](https://github.com/kubernetes/kops/pull/13401)
* Clean up channels and protokube targets [@olemarkus](https://github.com/olemarkus) [#13404](https://github.com/kubernetes/kops/pull/13404)
* Don't use bazel for dev-upload [@olemarkus](https://github.com/olemarkus) [#13407](https://github.com/kubernetes/kops/pull/13407)
* Clean up the rest of the dev-upload targets [@olemarkus](https://github.com/olemarkus) [#13408](https://github.com/kubernetes/kops/pull/13408)
* AWS-NODE-TERMINATION: Add possibility to set a tune image version [@anthonyhaussman](https://github.com/anthonyhaussman) [#13405](https://github.com/kubernetes/kops/pull/13405)
* Update netflify deps [@hakman](https://github.com/hakman) [#13412](https://github.com/kubernetes/kops/pull/13412)
* Promote alpha to stable [@MoShitrit](https://github.com/MoShitrit) [#13415](https://github.com/kubernetes/kops/pull/13415)
* Correctly detect GovCloud regions [@mixja](https://github.com/mixja) [#13410](https://github.com/kubernetes/kops/pull/13410)
* dev-upload from upload directory, not bazelupload [@olemarkus](https://github.com/olemarkus) [#13419](https://github.com/kubernetes/kops/pull/13419)
* fix member update when draining [@zetaab](https://github.com/zetaab) [#13414](https://github.com/kubernetes/kops/pull/13414)
* Update the security docs [@olemarkus](https://github.com/olemarkus) [#13421](https://github.com/kubernetes/kops/pull/13421)
* Remove id var, which is never used beyond that if test [@olemarkus](https://github.com/olemarkus) [#13420](https://github.com/kubernetes/kops/pull/13420)
* Pin actions to a full length commit SHA [@naveensrinivasan](https://github.com/naveensrinivasan) [#13395](https://github.com/kubernetes/kops/pull/13395)
* Digest images during remapping [@olemarkus](https://github.com/olemarkus) [#13422](https://github.com/kubernetes/kops/pull/13422)
* Do not return a '-1' exit if no keys found and json/yaml output [@hierynomus](https://github.com/hierynomus) [#13378](https://github.com/kubernetes/kops/pull/13378)
* Use non-bazel builds in kubetest2 [@olemarkus](https://github.com/olemarkus) [#13409](https://github.com/kubernetes/kops/pull/13409)
* Run CSI controller in masters [@zetaab](https://github.com/zetaab) [#13426](https://github.com/kubernetes/kops/pull/13426)
* Push ko-built images to staging [@olemarkus](https://github.com/olemarkus) [#13428](https://github.com/kubernetes/kops/pull/13428)
* Included githubactions in dependabot config [@naveensrinivasan](https://github.com/naveensrinivasan) [#13423](https://github.com/kubernetes/kops/pull/13423)
* Bump actions/setup-go from 2.2.0 to 3 [@dependabot[bot]](https://github.com/apps/dependabot),[@hakman](https://github.com/hakman) [#13429](https://github.com/kubernetes/kops/pull/13429)
* Use golang 1.18 for building [@olemarkus](https://github.com/olemarkus) [#13430](https://github.com/kubernetes/kops/pull/13430)
* Use golang 1.18.0 explicitly [@hakman](https://github.com/hakman) [#13432](https://github.com/kubernetes/kops/pull/13432)
* Install gcloud instead of gsutil [@hakman](https://github.com/hakman) [#13434](https://github.com/kubernetes/kops/pull/13434)
* Use the correct upload folder for the latest.txt file [@olemarkus](https://github.com/olemarkus) [#13435](https://github.com/kubernetes/kops/pull/13435)
* Fix upload dir in cloudbuild-artifacts target [@olemarkus](https://github.com/olemarkus) [#13437](https://github.com/kubernetes/kops/pull/13437)
* Skip upstream e2e log dump [@olemarkus](https://github.com/olemarkus) [#13438](https://github.com/kubernetes/kops/pull/13438)
* Remove direct dependency on klog v1 [@olemarkus](https://github.com/olemarkus) [#13447](https://github.com/kubernetes/kops/pull/13447)
* Bump semver for kubetest2 [@olemarkus](https://github.com/olemarkus) [#13446](https://github.com/kubernetes/kops/pull/13446)
* fix dns controller crashing for DO [@zak905](https://github.com/zak905) [#13443](https://github.com/kubernetes/kops/pull/13443)
* Add possibility to set PodAnnotations into NodeLocalDNS [@anthonyhaussman](https://github.com/anthonyhaussman) [#13396](https://github.com/kubernetes/kops/pull/13396)
* Disable CGO and enable trimpath [@olemarkus](https://github.com/olemarkus) [#13451](https://github.com/kubernetes/kops/pull/13451)
* Possibility to add additional routes in route tables of subnets [@guillomep](https://github.com/guillomep) [#13318](https://github.com/kubernetes/kops/pull/13318)
* Update containerd to v1.6.2 [@hakman](https://github.com/hakman) [#13455](https://github.com/kubernetes/kops/pull/13455)
* Don't run the CSI snapshot plugin if snapshot controller is not installed [@olemarkus](https://github.com/olemarkus) [#13453](https://github.com/kubernetes/kops/pull/13453)
* Add back hash for containerd v1.6.1 [@hakman](https://github.com/hakman) [#13462](https://github.com/kubernetes/kops/pull/13462)
* Use Cilium 1.11 as default [@olemarkus](https://github.com/olemarkus) [#12919](https://github.com/kubernetes/kops/pull/12919)
* Bump Cert Manager to 1.8 [@olemarkus](https://github.com/olemarkus) [#13464](https://github.com/kubernetes/kops/pull/13464)
* Pick the right OS server group when creating cloud groups [@ederst](https://github.com/ederst) [#13461](https://github.com/kubernetes/kops/pull/13461)
* Don't hash parts of the cluster name for e2e [@olemarkus](https://github.com/olemarkus) [#13354](https://github.com/kubernetes/kops/pull/13354)
* Enable etcd corruption check as mitigatio of 3.5 corruption issue [@olemarkus](https://github.com/olemarkus) [#13454](https://github.com/kubernetes/kops/pull/13454)
* Move Azure settings to cloudProvider.azure [@johngmyers](https://github.com/johngmyers) [#13065](https://github.com/kubernetes/kops/pull/13065)
* Add PDBs to addons where this was missing [@olemarkus](https://github.com/olemarkus) [#13475](https://github.com/kubernetes/kops/pull/13475)
* Bump NTH 1.16 and add excludeFromLoadBalancers option [@DingGGu](https://github.com/DingGGu) [#13467](https://github.com/kubernetes/kops/pull/13467)
* Move Openstack settings to cloudProvider.openstack [@johngmyers](https://github.com/johngmyers) [#13326](https://github.com/kubernetes/kops/pull/13326)
* Bump viper and cobra [@olemarkus](https://github.com/olemarkus) [#13482](https://github.com/kubernetes/kops/pull/13482)
* Bump cert-manager deps [@olemarkus](https://github.com/olemarkus) [#13481](https://github.com/kubernetes/kops/pull/13481)
* Remove explicit dependency on yamlv2 [@olemarkus](https://github.com/olemarkus) [#13483](https://github.com/kubernetes/kops/pull/13483)
* Use latest stable release by default for scenario test [@olemarkus](https://github.com/olemarkus) [#13476](https://github.com/kubernetes/kops/pull/13476)
* Remove GOPATH dependency for apimachinery [@hakman](https://github.com/hakman) [#13472](https://github.com/kubernetes/kops/pull/13472)
* Document NodeLocalDNS forwardToKubeDNS breaking change [@jorge07](https://github.com/jorge07) [#13448](https://github.com/kubernetes/kops/pull/13448)
* Fix typos in docs/getting_started/arguments [@scottchiang](https://github.com/scottchiang) [#13485](https://github.com/kubernetes/kops/pull/13485)
* Update kubetest2 [@olemarkus](https://github.com/olemarkus) [#13486](https://github.com/kubernetes/kops/pull/13486)
* Bump Ubuntu AMIs in alpha and stable [@olemarkus](https://github.com/olemarkus) [#13487](https://github.com/kubernetes/kops/pull/13487)
* Bump AWS CNI to version 1.10.3 [@MoShitrit](https://github.com/MoShitrit) [#13488](https://github.com/kubernetes/kops/pull/13488)
* Use aws_s3_object instead of deprecated aws_s3_bucket_object [@hakman](https://github.com/hakman) [#13491](https://github.com/kubernetes/kops/pull/13491)
* Docs getting started aws [@anthonytwh](https://github.com/anthonytwh) [#13489](https://github.com/kubernetes/kops/pull/13489)
* Update Calico and Canal to v3.21.5 [@hakman](https://github.com/hakman) [#13497](https://github.com/kubernetes/kops/pull/13497)
* Update to etcd-manager 3.0.20220417 [@justinsb](https://github.com/justinsb) [#13499](https://github.com/kubernetes/kops/pull/13499)
* Revert "Enable etcd corruption check" [@hakman](https://github.com/hakman) [#13495](https://github.com/kubernetes/kops/pull/13495)
* etcd 3 5 3 [@justinsb](https://github.com/justinsb) [#13501](https://github.com/kubernetes/kops/pull/13501)
* Remove bazel targets and tools [@olemarkus](https://github.com/olemarkus) [#13484](https://github.com/kubernetes/kops/pull/13484)
* Bump CCM 1.22 and 1.23 images to stable versions [@olemarkus](https://github.com/olemarkus) [#13506](https://github.com/kubernetes/kops/pull/13506)
* Use Cluster Autoscaler 1.23 for k8s 1.24 [@olemarkus](https://github.com/olemarkus) [#13510](https://github.com/kubernetes/kops/pull/13510)
* Adopt control-plane taint and remove master role labels [@olemarkus](https://github.com/olemarkus) [#13452](https://github.com/kubernetes/kops/pull/13452)
* Update aws-sdk-go to v1.43.41 [@hakman](https://github.com/hakman) [#13515](https://github.com/kubernetes/kops/pull/13515)
* Revert to using 1.23.0-alpha.0 for AWS CCM [@hakman](https://github.com/hakman) [#13514](https://github.com/kubernetes/kops/pull/13514)
* add cluster autoscaler pod annotations [@heybronson](https://github.com/heybronson) [#13511](https://github.com/kubernetes/kops/pull/13511)
* Trim GCE firewall rule names to their max length [@rifelpet](https://github.com/rifelpet) [#13513](https://github.com/kubernetes/kops/pull/13513)
* kubetest2-kops - Create ephemeral SSH keys [@rifelpet](https://github.com/rifelpet) [#13522](https://github.com/kubernetes/kops/pull/13522)
* Skip topology hints tests in k8s 1.23 [@rifelpet](https://github.com/rifelpet) [#13524](https://github.com/kubernetes/kops/pull/13524)
* Update remaining addon manifests for the control-plane node role [@rifelpet](https://github.com/rifelpet) [#13521](https://github.com/kubernetes/kops/pull/13521)
* Release notes for 1.22.5 [@olemarkus](https://github.com/olemarkus) [#13528](https://github.com/kubernetes/kops/pull/13528)
* Release notes for 1.23.1 [@olemarkus](https://github.com/olemarkus) [#13531](https://github.com/kubernetes/kops/pull/13531)
* Allow cluster autoscaler to read EC2 instance types to build catalog dynamically [@seh](https://github.com/seh) [#13532](https://github.com/kubernetes/kops/pull/13532)
* Use expected pointer type in type assertion when iterating over GS ACLs [@tesspib](https://github.com/tesspib) [#13534](https://github.com/kubernetes/kops/pull/13534)
* Shell out to ssh-keygen for creating ed25519 keys [@rifelpet](https://github.com/rifelpet) [#13538](https://github.com/kubernetes/kops/pull/13538)
* Update control plane toleration for external-dns [@rifelpet](https://github.com/rifelpet) [#13539](https://github.com/kubernetes/kops/pull/13539)
* Pin AWS CCM image tag for k8s 1.25 [@rifelpet](https://github.com/rifelpet) [#13543](https://github.com/kubernetes/kops/pull/13543)
* fix pod annotations in addon yamls [@heybronson](https://github.com/heybronson) [#13536](https://github.com/kubernetes/kops/pull/13536)
* Skip SCTP tests in k8s 1.25 as well [@rifelpet](https://github.com/rifelpet) [#13545](https://github.com/kubernetes/kops/pull/13545)
* Bump k8s deps to 1.24.0-rc.0 [@olemarkus](https://github.com/olemarkus) [#13548](https://github.com/kubernetes/kops/pull/13548)
* Trim GCE Subnet and Disk names [@rifelpet](https://github.com/rifelpet) [#13546](https://github.com/kubernetes/kops/pull/13546)
* Update codegen to v1.24.0-rc.0 [@hakman](https://github.com/hakman) [#13549](https://github.com/kubernetes/kops/pull/13549)
* Update Go to v1.18.1 [@hakman](https://github.com/hakman) [#13550](https://github.com/kubernetes/kops/pull/13550)
* Upgrade flannel to 0.17.0 [@rifelpet](https://github.com/rifelpet) [#13552](https://github.com/kubernetes/kops/pull/13552)
* Always set cluster-id flag for Protokube [@hakman](https://github.com/hakman) [#13555](https://github.com/kubernetes/kops/pull/13555)
* Creaate kubeconfig for the root user on Ubuntu [@hakman](https://github.com/hakman) [#13553](https://github.com/kubernetes/kops/pull/13553)
* Bump alpha channel with april releases [@olemarkus](https://github.com/olemarkus) [#13556](https://github.com/kubernetes/kops/pull/13556)
* Enable internal load balancers when running on GCP [@nat-henderson](https://github.com/nat-henderson) [#13445](https://github.com/kubernetes/kops/pull/13445)
* Add back support for Ubuntu 18.04 [@hakman](https://github.com/hakman) [#13557](https://github.com/kubernetes/kops/pull/13557)
* Add support for Rocky Linux 8 [@hakman](https://github.com/hakman) [#13559](https://github.com/kubernetes/kops/pull/13559)
* Re-add net.bridge settings for flannel [@rifelpet](https://github.com/rifelpet) [#13564](https://github.com/kubernetes/kops/pull/13564)
* Update Canal's Flannel to v0.15.1 [@tesspib](https://github.com/tesspib) [#13562](https://github.com/kubernetes/kops/pull/13562)
* Bump AWS CNI to version 1.11.0 [@MoShitrit](https://github.com/MoShitrit) [#13493](https://github.com/kubernetes/kops/pull/13493)
* Fix OIDC Provider cleanup [@rifelpet](https://github.com/rifelpet) [#13571](https://github.com/kubernetes/kops/pull/13571)
* Include sysctls in toolbox dump [@rifelpet](https://github.com/rifelpet) [#13570](https://github.com/kubernetes/kops/pull/13570)
* Add NodeTerminationHandler CPU request settings [@glebiller](https://github.com/glebiller) [#13566](https://github.com/kubernetes/kops/pull/13566)
* Add docs for Rocky Linux 8 [@hakman](https://github.com/hakman) [#13561](https://github.com/kubernetes/kops/pull/13561)
* Don't digest side-loaded images [@hakman](https://github.com/hakman) [#13576](https://github.com/kubernetes/kops/pull/13576)
* Update containerd to v1.6.3 [@hakman](https://github.com/hakman) [#13578](https://github.com/kubernetes/kops/pull/13578)
* Upgrade EBS CSI Driver to v1.6.0 [@rifelpet](https://github.com/rifelpet) [#13565](https://github.com/kubernetes/kops/pull/13565)
* Bump nvidia device plugin to 0.11 and nvidia driver to 510 [@olemarkus](https://github.com/olemarkus) [#13580](https://github.com/kubernetes/kops/pull/13580)
* Make kOps version show server version too [@olemarkus](https://github.com/olemarkus) [#12327](https://github.com/kubernetes/kops/pull/12327)
* Spotinst: Add support for EBS root volume encryption [@liranp](https://github.com/liranp) [#13585](https://github.com/kubernetes/kops/pull/13585)
* promote alpha to stable (k8s versions bump) [@MoShitrit](https://github.com/MoShitrit) [#13587](https://github.com/kubernetes/kops/pull/13587)
* Remove remaining bazel references [@rifelpet](https://github.com/rifelpet) [#13586](https://github.com/kubernetes/kops/pull/13586)
* Revert containerd v1.6.3 upgrade [@rifelpet](https://github.com/rifelpet) [#13582](https://github.com/kubernetes/kops/pull/13582)
* Update etcd to v3.5.4 [@hakman](https://github.com/hakman) [#13588](https://github.com/kubernetes/kops/pull/13588)
* Bump cilium to 1.11.4 [@olemarkus](https://github.com/olemarkus) [#13581](https://github.com/kubernetes/kops/pull/13581)
* Update golangci-lint to v1.45.2 [@hakman](https://github.com/hakman) [#13584](https://github.com/kubernetes/kops/pull/13584)
* Don't install nfs-common/nfs-utils by default [@hakman](https://github.com/hakman) [#13577](https://github.com/kubernetes/kops/pull/13577)
* doc(mfa): add another workaround using aws-vault [@techieforfun](https://github.com/techieforfun) [#13591](https://github.com/kubernetes/kops/pull/13591)
* add priorities to CSI openstack [@zetaab](https://github.com/zetaab) [#13594](https://github.com/kubernetes/kops/pull/13594)
* Fix unexpected type for object metadata when using gossip DNS [@hakman](https://github.com/hakman) [#13592](https://github.com/kubernetes/kops/pull/13592)
* fix immutable retry logic when updating member [@zetaab](https://github.com/zetaab) [#13595](https://github.com/kubernetes/kops/pull/13595)
* Add support for installing additional packages [@hakman](https://github.com/hakman) [#13590](https://github.com/kubernetes/kops/pull/13590)
* Update containerd to v1.6.4 [@hakman](https://github.com/hakman) [#13596](https://github.com/kubernetes/kops/pull/13596)
* Update etcd-manager to v3.0.20220503 [@hakman](https://github.com/hakman) [#13598](https://github.com/kubernetes/kops/pull/13598)
* Bump k8s.io deps to 1.24.0 [@olemarkus](https://github.com/olemarkus) [#13601](https://github.com/kubernetes/kops/pull/13601)
* update deps [@zetaab](https://github.com/zetaab) [#13602](https://github.com/kubernetes/kops/pull/13602)
* update OpenStack CSI & cloud-controller manifest [@zetaab](https://github.com/zetaab) [#13603](https://github.com/kubernetes/kops/pull/13603)
* Add hashes for containerd and Docker in order to fix CVE-2022-23648 [@drequena](https://github.com/drequena) [#13606](https://github.com/kubernetes/kops/pull/13606)
* Add IRSA for kube-router [@rifelpet](https://github.com/rifelpet) [#13609](https://github.com/kubernetes/kops/pull/13609)
* Avoid "/etc/resolv.conf" file loopback for Flatcar Container Linux distribution [@seh](https://github.com/seh) [#13617](https://github.com/kubernetes/kops/pull/13617)
* Skip in-tree volume limits test [@rifelpet](https://github.com/rifelpet) [#13621](https://github.com/kubernetes/kops/pull/13621)
* Add initial support for Hetzner Cloud [@hakman](https://github.com/hakman) [#13572](https://github.com/kubernetes/kops/pull/13572)
* remove bind address for occm [@zetaab](https://github.com/zetaab) [#13623](https://github.com/kubernetes/kops/pull/13623)
* Add docs for the packages option [@hakman](https://github.com/hakman) [#13622](https://github.com/kubernetes/kops/pull/13622)
* Update Calico and Canal to v3.23.0 [@hakman](https://github.com/hakman) [#13625](https://github.com/kubernetes/kops/pull/13625)
* Use latest staging AWS CCM for k8s 1.24+ [@hakman](https://github.com/hakman) [#13626](https://github.com/kubernetes/kops/pull/13626)
* Allow returning multiple addresses for API [@hakman](https://github.com/hakman) [#13624](https://github.com/kubernetes/kops/pull/13624)
* [DigitalOcean]: make droplet image constant when building droplet in for fitask [@zak905](https://github.com/zak905) [#13628](https://github.com/kubernetes/kops/pull/13628)
* Upgrade Karpenter to 0.10.0 [@rifelpet](https://github.com/rifelpet) [#13547](https://github.com/kubernetes/kops/pull/13547)
* Add load balancer support for Hetzner [@hakman](https://github.com/hakman) [#13629](https://github.com/kubernetes/kops/pull/13629)

## 1.24.0-alpha.4 to 1.24.0-alpha.5

* Release notes for 1.23.2 [@olemarkus](https://github.com/olemarkus) [#13632](https://github.com/kubernetes/kops/pull/13632)
* Release notes for 1.24.0-alpha.4 [@hakman](https://github.com/hakman) [#13634](https://github.com/kubernetes/kops/pull/13634)
* Use fixed staging AWS CCM for k8s 1.24+ [@hakman](https://github.com/hakman) [#13637](https://github.com/kubernetes/kops/pull/13637)
* Update runc to v1.1.2 [@hakman](https://github.com/hakman) [#13638](https://github.com/kubernetes/kops/pull/13638)
* add annotation support to aws-ebs-csi pods [@avish42](https://github.com/avish42) [#13600](https://github.com/kubernetes/kops/pull/13600)
* Bump cluster autoscaler image for 1.24 [@olemarkus](https://github.com/olemarkus) [#13642](https://github.com/kubernetes/kops/pull/13642)
* Ignore digest for kube-apiserver-healthcheck model test [@hakman](https://github.com/hakman) [#13639](https://github.com/kubernetes/kops/pull/13639)
* Update documentation for manageStorageClasses [@minkimipt](https://github.com/minkimipt) [#13641](https://github.com/kubernetes/kops/pull/13641)
* fix typo in docs/instance-groups.md [@marquezika](https://github.com/marquezika) [#13635](https://github.com/kubernetes/kops/pull/13635)
* Fix code block in manageStorageClasses docs [@minkimipt](https://github.com/minkimipt) [#13645](https://github.com/kubernetes/kops/pull/13645)
* [Digital Ocean] Add CSI driver for DO block storage [@srikiz](https://github.com/srikiz) [#13643](https://github.com/kubernetes/kops/pull/13643)
* Update AWS CCM to 13.05.2022 releases [@hakman](https://github.com/hakman) [#13644](https://github.com/kubernetes/kops/pull/13644)
* Skip nfs tests in all scenarios [@hakman](https://github.com/hakman) [#13648](https://github.com/kubernetes/kops/pull/13648)
* Tweak integration tests to be simpler [@justinsb](https://github.com/justinsb) [#13647](https://github.com/kubernetes/kops/pull/13647)
* Use Calico v3.21 with older versions of k8s [@hakman](https://github.com/hakman) [#13649](https://github.com/kubernetes/kops/pull/13649)
* Use -ginkgo.junit-report instead of -ginkgo.reportFile [@hakman](https://github.com/hakman) [#13650](https://github.com/kubernetes/kops/pull/13650)
* Allow editing clusters with Hetzner [@hakman](https://github.com/hakman) [#13654](https://github.com/kubernetes/kops/pull/13654)
* Bump actions/setup-go from 3.0.0 to 3.1.0 [@dependabot[bot]](https://github.com/apps/dependabot) [#13653](https://github.com/kubernetes/kops/pull/13653)
* Promote May AMIs in channels [@olemarkus](https://github.com/olemarkus) [#13658](https://github.com/kubernetes/kops/pull/13658)
* Allow the AWS Load Balancer Controller to use WAFs [@seh](https://github.com/seh) [#13636](https://github.com/kubernetes/kops/pull/13636)
* Update controller-runtime to v0.12.0 [@hakman](https://github.com/hakman) [#13655](https://github.com/kubernetes/kops/pull/13655)
* Set nodeSelector and command to null [@zetaab](https://github.com/zetaab) [#13656](https://github.com/kubernetes/kops/pull/13656)
* Add a nameservers parameter for cert-manager. [@jim-barber-he](https://github.com/jim-barber-he) [#13567](https://github.com/kubernetes/kops/pull/13567)
* Allow overriding the kubernetes version when upgrading the cluster [@hakman](https://github.com/hakman) [#13652](https://github.com/kubernetes/kops/pull/13652)
* Bump EBS CSI driver to 1.6.1 [@olemarkus](https://github.com/olemarkus) [#13664](https://github.com/kubernetes/kops/pull/13664)
* Bump Load Balancer Controller to 2.4.1 [@olemarkus](https://github.com/olemarkus) [#13665](https://github.com/kubernetes/kops/pull/13665)
* Bump Cilium to 1.11.5 [@olemarkus](https://github.com/olemarkus) [#13666](https://github.com/kubernetes/kops/pull/13666)
* Add missing backslash to aws.md [@piec](https://github.com/piec) [#13669](https://github.com/kubernetes/kops/pull/13669)
* Bump NTH to 1.16.4 and add support for scheduled instance change events [@olemarkus](https://github.com/olemarkus) [#13662](https://github.com/kubernetes/kops/pull/13662)
* Bump EBS CSI driver to 1.6.2 [@olemarkus](https://github.com/olemarkus) [#13670](https://github.com/kubernetes/kops/pull/13670)
* Bump metrics-server to 0.6.1 [@olemarkus](https://github.com/olemarkus) [#13674](https://github.com/kubernetes/kops/pull/13674)
* Bump external-dns to 0.11.0 [@olemarkus](https://github.com/olemarkus) [#13673](https://github.com/kubernetes/kops/pull/13673)

## 1.24.0-alpha.5 to 1.24.0-beta.1

* Release notes for 1.24.0-alpha.5 [@olemarkus](https://github.com/olemarkus) [#13676](https://github.com/kubernetes/kops/pull/13676)
* Bump coredns to 1.8.6 [@olemarkus](https://github.com/olemarkus) [#13677](https://github.com/kubernetes/kops/pull/13677)
* Add Support for OVN Load Balancer for OpenStack [@ching-kuo](https://github.com/ching-kuo),[@zetaab](https://github.com/zetaab) [#13678](https://github.com/kubernetes/kops/pull/13678)
* Add release note and getting started entries for Hetzner [@hakman](https://github.com/hakman) [#13680](https://github.com/kubernetes/kops/pull/13680)
* Fix kops update for OpenStack with LB [@ching-kuo](https://github.com/ching-kuo) [#13682](https://github.com/kubernetes/kops/pull/13682)
* Update Calico and Canal to v3.23.1 [@hakman](https://github.com/hakman) [#13672](https://github.com/kubernetes/kops/pull/13672)
* bump alpha k8s versions with May releases [@MoShitrit](https://github.com/MoShitrit) [#13683](https://github.com/kubernetes/kops/pull/13683)
* Update troubleshoot.md [@simonccc](https://github.com/simonccc) [#13685](https://github.com/kubernetes/kops/pull/13685)
* Add support for configuring which metrics cilium will export [@olemarkus](https://github.com/olemarkus) [#13684](https://github.com/kubernetes/kops/pull/13684)
* Remove unused DNS logic from Protokube [@hakman](https://github.com/hakman) [#13689](https://github.com/kubernetes/kops/pull/13689)
* Fix Protokube gossip flag [@hakman](https://github.com/hakman) [#13692](https://github.com/kubernetes/kops/pull/13692)
* Bump actions/setup-go from 3.1.0 to 3.2.0 [@dependabot[bot]](https://github.com/apps/dependabot) [#13694](https://github.com/kubernetes/kops/pull/13694)
* Refactor cloud providers and remove unused code from Protokube [@hakman](https://github.com/hakman) [#13691](https://github.com/kubernetes/kops/pull/13691)
* Use dependabot for Go deps [@hakman](https://github.com/hakman) [#13696](https://github.com/kubernetes/kops/pull/13696)
* Fix links to go docs [@diversario](https://github.com/diversario) [#13705](https://github.com/kubernetes/kops/pull/13705)
* Update gophercloud to v0.25.0 [@ching-kuo](https://github.com/ching-kuo) [#13710](https://github.com/kubernetes/kops/pull/13710)
* Use build tags for Protokube [@hakman](https://github.com/hakman) [#13706](https://github.com/kubernetes/kops/pull/13706)
* Clarify difference between terraform and kOps state stores [@hakman](https://github.com/hakman) [#13709](https://github.com/kubernetes/kops/pull/13709)
* Bump node termination handler to 1.16.5 [@olemarkus](https://github.com/olemarkus) [#13711](https://github.com/kubernetes/kops/pull/13711)
* promote alpha to stable (k8s releases) [@MoShitrit](https://github.com/MoShitrit) [#13713](https://github.com/kubernetes/kops/pull/13713)
* Migrate EBS CSI images back to registry.k8s.io [@rifelpet](https://github.com/rifelpet) [#13718](https://github.com/kubernetes/kops/pull/13718)
* Bump github.com/prometheus/client_golang from 1.12.1 to 1.12.2 [@dependabot[bot]](https://github.com/apps/dependabot) [#13697](https://github.com/kubernetes/kops/pull/13697)
* update openstack ccm + csi [@zetaab](https://github.com/zetaab) [#13716](https://github.com/kubernetes/kops/pull/13716)
* Bump github.com/spotinst/spotinst-sdk-go from 1.118.0 to 1.120.0 [@dependabot[bot]](https://github.com/apps/dependabot) [#13699](https://github.com/kubernetes/kops/pull/13699)
* Bump aws cni to 1.11.2 [@MoShitrit](https://github.com/MoShitrit) [#13726](https://github.com/kubernetes/kops/pull/13726)
* Add CSI driver for Hetzner [@hakman](https://github.com/hakman) [#13728](https://github.com/kubernetes/kops/pull/13728)
* Use kubectl replace instead of apply when updating addons [@olemarkus](https://github.com/olemarkus) [#13731](https://github.com/kubernetes/kops/pull/13731)

## 1.24.0-beta.1 to 1.24.0-beta.2

* Channels to have exit status 1 on apply failure [@olemarkus](https://github.com/olemarkus) [#13749](https://github.com/kubernetes/kops/pull/13749)
* Add support for setting mode field on file assets [@yurrriq](https://github.com/yurrriq) [#13715](https://github.com/kubernetes/kops/pull/13715)
* Revert "Use kubectl replace instead of apply when updating addons" [@hakman](https://github.com/hakman) [#13761](https://github.com/kubernetes/kops/pull/13761)
* Fix API group name for ingresses in DNS Controller [@julienperignon](https://github.com/julienperignon) [#13750](https://github.com/kubernetes/kops/pull/13750)
* Update containerd and Docker versions [@hakman](https://github.com/hakman) [#13741](https://github.com/kubernetes/kops/pull/13741)
* Update AWS CCM images for k8s 1.20-1.22 [@hakman](https://github.com/hakman) [#13748](https://github.com/kubernetes/kops/pull/13748)
* Update runc to v1.1.3 [@hakman](https://github.com/hakman) [#13763](https://github.com/kubernetes/kops/pull/13763)
* Fix namespace for cert manager webhook config [@olemarkus](https://github.com/olemarkus) [#13773](https://github.com/kubernetes/kops/pull/13773)
* Add back the metrics-server 443 port with a new name [@olemarkus](https://github.com/olemarkus) [#13779](https://github.com/kubernetes/kops/pull/13779)
* Avoid spurious changes with ed25519 keys [@hakman](https://github.com/hakman) [#13774](https://github.com/kubernetes/kops/pull/13774)
* Fix broken node selector for node termination handler [@olemarkus](https://github.com/olemarkus) [#13781](https://github.com/kubernetes/kops/pull/13781)

## 1.24.0-beta.2 to 1.24.0-beta.3

* Replace flexdriver with busybox [@zetaab](https://github.com/zetaab) [#13809](https://github.com/kubernetes/kops/pull/13809)
* Update etcd-manager to v3.0.20220617 [@hakman](https://github.com/hakman) [#13824](https://github.com/kubernetes/kops/pull/13824)
* Do not run cluster autoscaler on spot instances [@olemarkus](https://github.com/olemarkus) [#13846](https://github.com/kubernetes/kops/pull/13846)
* Fix GCE resource tracking [@hakman](https://github.com/hakman) [#13857](https://github.com/kubernetes/kops/pull/13857)
* Adding GuestAccelerators to InstanceTemplate [@jonasasx](https://github.com/jonasasx) [#13707](https://github.com/kubernetes/kops/pull/13707)
* Limit GCE tag for role to 63 chars [@hakman](https://github.com/hakman) [#13866](https://github.com/kubernetes/kops/pull/13866)
* Replace manifests after apply [@olemarkus](https://github.com/olemarkus) [#13819](https://github.com/kubernetes/kops/pull/13819)
* Don't try to manage the kube-system namespace [@hakman](https://github.com/hakman) [#13764](https://github.com/kubernetes/kops/pull/13764)
* Release 1.24.0-beta.3 [@olemarkus](https://github.com/olemarkus) [#13880](https://github.com/kubernetes/kops/pull/13880)

## 1.24.0-beta.3 to 1.24.0

* Clean-up firewall rules that contain targets with the cluster name hash [@hakman](https://github.com/hakman) [#13869](https://github.com/kubernetes/kops/pull/13869)
* Log errors from detachInstance [@olemarkus](https://github.com/olemarkus) [#13896](https://github.com/kubernetes/kops/pull/13896)
* gce: set ProvisioningModel on InstanceTemplate [@justinsb](https://github.com/justinsb) [#13902](https://github.com/kubernetes/kops/pull/13902)
* gce: Move out of beta, drop feature flag [@justinsb](https://github.com/justinsb),[@hakman](https://github.com/hakman) [#13903](https://github.com/kubernetes/kops/pull/13903)
* Fix cleanup of firewall rules that contain targets with the cluster name hash [@hakman](https://github.com/hakman) [#13907](https://github.com/kubernetes/kops/pull/13907)
* Only fail an addon update if the final apply fails. Install PKI anyway [@olemarkus](https://github.com/olemarkus) [#13897](https://github.com/kubernetes/kops/pull/13897)
* Update dependencies for 1.24 [@hakman](https://github.com/hakman) [#13919](https://github.com/kubernetes/kops/pull/13919)
* gce: Refactor ClusterPrefixedName and ClusterSuffixedName to not return error [@hakman](https://github.com/hakman) [#13920](https://github.com/kubernetes/kops/pull/13920)
* Make it possible to enable the shield addon for LBC [@olemarkus](https://github.com/olemarkus) [#13929](https://github.com/kubernetes/kops/pull/13929)
* Make IRSA webhook configure apps to use regional STS and set the default region on them [@olemarkus](https://github.com/olemarkus) [#13926](https://github.com/kubernetes/kops/pull/13926)
* Limit GCE names to 63 chars for various resources [@hakman](https://github.com/hakman) [#13873](https://github.com/kubernetes/kops/pull/13873)
* Mount /etc/hosts from host for CoreDNS [@hakman](https://github.com/hakman) [#13922](https://github.com/kubernetes/kops/pull/13922)
* Limit GCE router name to 63 chars [@hakman](https://github.com/hakman) [#13932](https://github.com/kubernetes/kops/pull/13932)
* Update Cilium to 1.11.6 [@ReillyBrogan](https://github.com/ReillyBrogan) [#13917](https://github.com/kubernetes/kops/pull/13917)
* Remove the v1alpha3 API version [@johngmyers](https://github.com/johngmyers) [#13935](https://github.com/kubernetes/kops/pull/13935)
* Fix unsetting ASG max price [@olemarkus](https://github.com/olemarkus) [#13852](https://github.com/kubernetes/kops/pull/13852)
* Revert "Add back the metrics-server 443 port with a new name" [@olemarkus](https://github.com/olemarkus) [#13940](https://github.com/kubernetes/kops/pull/13940)
* aws: introduce maximum instance lifetime in cluster [@sterchelen](https://github.com/sterchelen) [#13892](https://github.com/kubernetes/kops/pull/13892)
* Be more specific when filtering OS instance ports [@ederst](https://github.com/ederst) [#13861](https://github.com/kubernetes/kops/pull/13861)
* Use csi-snapshotter for OS only when the controller is enabled [@ederst](https://github.com/ederst) [#13890](https://github.com/kubernetes/kops/pull/13890)
* Bump EBS CSI driver to 1.8.0 [@hakman](https://github.com/hakman) [#13939](https://github.com/kubernetes/kops/pull/13939)

## 1.24.0 to 1.24.1

* Use Calico v3.23 for Kubernetes 1.22+ [@hakman](https://github.com/hakman) [#13901](https://github.com/kubernetes/kops/pull/13901)
* Fix AWS IAM Authenticator nodeSelector in k8s 1.24  [@rifelpet](https://github.com/rifelpet) [#13965](https://github.com/kubernetes/kops/pull/13965)
* Skip deregistering the instance during rolling update for Spotinst [@hakman](https://github.com/hakman) [#13970](https://github.com/kubernetes/kops/pull/13970)
* Upgrade aws-iam-authenticator to v0.5.9 [@rifelpet](https://github.com/rifelpet) [#13979](https://github.com/kubernetes/kops/pull/13979)
* Use only IPv4 for Hetzner servers [@hakman](https://github.com/hakman) [#13982](https://github.com/kubernetes/kops/pull/13982)
* Add option to set etcd-manager backup interval [@hakman](https://github.com/hakman) [#13975](https://github.com/kubernetes/kops/pull/13975)
* Update etcd-manager to v3.0.20220717 [@hakman](https://github.com/hakman) [#13990](https://github.com/kubernetes/kops/pull/13990)
* Update Go to v1.18.4 [@hakman](https://github.com/hakman) [#13994](https://github.com/kubernetes/kops/pull/13994)
* Add option to set number of replicas for pod-identity-webhook [@hakman](https://github.com/hakman) [#13986](https://github.com/kubernetes/kops/pull/13986)
* Upgrade DO CSI controller to 4.2.0 [@rifelpet](https://github.com/rifelpet) [#14005](https://github.com/kubernetes/kops/pull/14005)
* Update k8s.io/client-go to match k8s.io/api [@hakman](https://github.com/hakman) [#14003](https://github.com/kubernetes/kops/pull/14003)
* Switch to latest MacOS version for CI [@hakman](https://github.com/hakman) [#14015](https://github.com/kubernetes/kops/pull/14015)
* Revert to using instance private DNS name to lookup hostname [@hakman](https://github.com/hakman) [#14024](https://github.com/kubernetes/kops/pull/14024)
* Add server group management for Hetzner [@hakman](https://github.com/hakman) [#14018](https://github.com/kubernetes/kops/pull/14018)
* Update dependencies for kOps 1.24 [@hakman](https://github.com/hakman) [#13989](https://github.com/kubernetes/kops/pull/13989)
* Update Calico to v3.23.2 [@hakman](https://github.com/hakman) [#13908](https://github.com/kubernetes/kops/pull/13908)
* Update etcd-manager to v3.0.20220727 [@hakman](https://github.com/hakman) [#14038](https://github.com/kubernetes/kops/pull/14038)
*  Check keyset existence before attempting to distrust [@yurrriq](https://github.com/yurrriq) [#14041](https://github.com/kubernetes/kops/pull/14041)
* Fix SIGSEGV when deleting a Hetzner instance [@hakman](https://github.com/hakman) [#14046](https://github.com/kubernetes/kops/pull/14046)
* Remove namespaces from cluster-scoped resources in CNI manifests [@rifelpet](https://github.com/rifelpet) [#14053](https://github.com/kubernetes/kops/pull/14053)
* Enable rolling updates for Hetzner [@hakman](https://github.com/hakman) [#14034](https://github.com/kubernetes/kops/pull/14034)
* Use cabundle for etcd CA files to fix key rotation in HA clusters [@olemarkus](https://github.com/olemarkus) [#14054](https://github.com/kubernetes/kops/pull/14054)

## 1.24.1 to 1.24.2

* Add config drive as a source for OpenStack instance metadata [@ederst](https://github.com/ederst) [#13845](https://github.com/kubernetes/kops/pull/13845)
* Allow configuring OpenStack CCM networking options [@ederst](https://github.com/ederst) [#14017](https://github.com/kubernetes/kops/pull/14017)
* aws-ebs-csi-driver: remove preStop hook [@sterchelen](https://github.com/sterchelen) [#14081](https://github.com/kubernetes/kops/pull/14081)
* Add option to configure runc version for containerd [@hakman](https://github.com/hakman) [#14090](https://github.com/kubernetes/kops/pull/14090)
* Bump nvidia device plugin to 0.12.0 [@ddelange](https://github.com/ddelange) [#13745](https://github.com/kubernetes/kops/pull/13745)
* Add hashes for containerd v1.6.7 [@hakman](https://github.com/hakman) [#14093](https://github.com/kubernetes/kops/pull/14093)
* Add deployment-specific selectors to nth pdb [@olemarkus](https://github.com/olemarkus) [#14113](https://github.com/kubernetes/kops/pull/14113)
* Disable some flags in kube-controller-manager and kube-scheduler when logging-format is not text [@h3poteto](https://github.com/h3poteto) [#14115](https://github.com/kubernetes/kops/pull/14115)
* Limit GCE network names to 63 chars [@hakman](https://github.com/hakman) [#14134](https://github.com/kubernetes/kops/pull/14134)
* Bump the CCM images [@olemarkus](https://github.com/olemarkus) [#14130](https://github.com/kubernetes/kops/pull/14130)
* Update runc to v1.1.4 [@hakman](https://github.com/hakman) [#14188](https://github.com/kubernetes/kops/pull/14188)
* OIDC: Tolerate extra service-account key set items [@seh](https://github.com/seh) [#14175](https://github.com/kubernetes/kops/pull/14175)
* Bump cilium to 1.11.8 [@olemarkus](https://github.com/olemarkus) [#14137](https://github.com/kubernetes/kops/pull/14137)
* Bump cert-manager to 1.8.2 [@olemarkus](https://github.com/olemarkus) [#14212](https://github.com/kubernetes/kops/pull/14212)
* Calico: Work around host port/conntrack problem [@seh](https://github.com/seh) [#14205](https://github.com/kubernetes/kops/pull/14205)

## 1.24.2 to 1.24.3

* aws-node-termination-handler to match node using providerID instead of AWS DNS name [@anthonyhaussman](https://github.com/anthonyhaussman) [#14244](https://github.com/kubernetes/kops/pull/14244)
* AWS LBC needs ec2:DescribeVpcPeeringConnections for IPv6 [@johngmyers](https://github.com/johngmyers) [#14255](https://github.com/kubernetes/kops/pull/14255)
* Ignore the _rundir that kubetest2 now creates [@olemarkus](https://github.com/olemarkus) [#13914](https://github.com/kubernetes/kops/pull/13914)
* Fix openstack tag limitation [@akkina2107](https://github.com/akkina2107) [#13853](https://github.com/kubernetes/kops/pull/13853)
* Warm pool-enabled ASGs scaled to zero will no longer panic [@olemarkus](https://github.com/olemarkus) [#14251](https://github.com/kubernetes/kops/pull/14251)
* Bump AWS CNI to 1.11.3 [@MoShitrit](https://github.com/MoShitrit) [#14107](https://github.com/kubernetes/kops/pull/14107)

## 1.24.3 to 1.24.4

* Avoid spurious changes with NLB due to access log config [@hakman](https://github.com/hakman) [#14319](https://github.com/kubernetes/kops/pull/14319)
* Add --network-id alias for --vpc flag [@hakman](https://github.com/hakman) [#14326](https://github.com/kubernetes/kops/pull/14326)
* Avoid spurious changes with bastion hosts due to user data [@hakman](https://github.com/hakman) [#14318](https://github.com/kubernetes/kops/pull/14318)
* cluster-autoscaler : Add iam permission autoscaling:DescribeScalingActivities needed since 1.24 version [@noony](https://github.com/noony) [#14317](https://github.com/kubernetes/kops/pull/14317)
* Disable rp_filter on cilium hosts [@olemarkus](https://github.com/olemarkus) [#14369](https://github.com/kubernetes/kops/pull/14369)
* gce: memberlist needs TCP also [@justinsb](https://github.com/justinsb) [#14364](https://github.com/kubernetes/kops/pull/14364)
* update node-problem-detector to version 0.8.12 [@jjinno](https://github.com/jjinno) [#14382](https://github.com/kubernetes/kops/pull/14382)
* update k8s cloudprovider openstack images [@zetaab](https://github.com/zetaab) [#14401](https://github.com/kubernetes/kops/pull/14401)
* Fix default launch spec detection for spotinst [@2solt](https://github.com/2solt) [#14389](https://github.com/kubernetes/kops/pull/14389)

## 1.24.4 to 1.24.5

* Update containerd to v1.6.9 [@hakman](https://github.com/hakman) [#14458](https://github.com/kubernetes/kops/pull/14458)
* Update Calico and Canal to v3.24.3 [@hakman](https://github.com/hakman) [#14466](https://github.com/kubernetes/kops/pull/14466)
* Set customizable affinity and tolerations for coredns-autoscaler to match main coredns deployment [@MoShitrit](https://github.com/MoShitrit) [#14503](https://github.com/kubernetes/kops/pull/14503)
* AWS CNI template updates [@MoShitrit](https://github.com/MoShitrit) [#14513](https://github.com/kubernetes/kops/pull/14513)
* Update Go to v1.18.8 [@hakman](https://github.com/hakman) [#14555](https://github.com/kubernetes/kops/pull/14555)
* Update Calico and Canal to latest versions [@hakman](https://github.com/hakman) [#14558](https://github.com/kubernetes/kops/pull/14558)
* Update containerd to v1.6.10 [@hakman](https://github.com/hakman) [#14550](https://github.com/kubernetes/kops/pull/14550)
* use sprig join for template functions [@heybronson](https://github.com/heybronson) [#14564](https://github.com/kubernetes/kops/pull/14564)
* aws: Fix SIGSEGV when using instance selector [@hakman](https://github.com/hakman) [#14576](https://github.com/kubernetes/kops/pull/14576)
* Add generics alternatives for fi.Bool/Float*/Int*/String*() [@hakman](https://github.com/hakman) [#14595](https://github.com/kubernetes/kops/pull/14595)
* Remove CloudFormation tests [@johngmyers](https://github.com/johngmyers) [#14602](https://github.com/kubernetes/kops/pull/14602)
* Fix pdb for identity webhook [@olemarkus](https://github.com/olemarkus) [#14442](https://github.com/kubernetes/kops/pull/14442)
* aws: Add `ec2:DescribeAvailabilityZones` to the AWS CCM permissions list [@lobziik](https://github.com/lobziik) [#14650](https://github.com/kubernetes/kops/pull/14650)
* aws: Limit the number of target groups updated per operation [@hakman](https://github.com/hakman) [#14648](https://github.com/kubernetes/kops/pull/14648)
* gce: Allow Cilium to connect to its etcd cluster [@hakman](https://github.com/hakman) [#14655](https://github.com/kubernetes/kops/pull/14655)