37 KiB

Raw Blame History

Release notes for kOps 1.24 series

⚠ kOps 1.24 has not been released yet! ⚠

This is a document to gather the release notes prior to the release.

Significant changes

Karpenter support

By enabling the Karpenter feature flag, users can now create InstanceGroups managed by (https://karpenter.sh)[Karpenter]:

spec:
  manager: Karpenter

You can also start a Karpenter-only cluster with kops create cluster --instance-manager=karpenter ...

kOps will directly manage the Karpenter Provisioner resources. Read more about how Karpenter works on kOps in the Karpenter docs.

Other significant changes

The minimum version for the Terraform AWS Provider has been bumped to 4.0.0 to address the deprecation of the aws_s3_bucket_object resource and its replacement with the aws_s3_object resource. Such resources will be destroyed and recreated without downtime when applying the changes.

Breaking changes

Support for Kubernetes version 1.18 has been removed.
Support for Aliyun/Alibaba Cloud has been removed.
Support for Docker has been removed for Kubernetes 1.24+. See https://kubernetes.io/blog/2020/12/02/dockershim-faq
Cert Manager upgraded from 1.6 to 1.8. This has backwards-breaking changes. See upgrading from 1.6 to 1.7 and [1.1.7 to 1.8.

Required actions

Deprecations

Support for Kubernetes version 1.19 is deprecated and will be removed in kOps 1.25.
Support for Kubernetes version 1.20 is deprecated and will be removed in kOps 1.26.
All legacy addons are deprecated in favor of managed addons, including the metrics server addon and the autoscaler addon.
The node-role.kubernetes.io/master and kubernetes.io/role labels are deprecated and might be removed from control plane nodes in future versions of kOps.
Due to lack of maintainers, the CloudFormation support has been deprecated. The current implementation will be left as-is until the implementation needs updates or otherwise becomes incompatible. At that point, it will be removed. We very much welcome anyone willing to contribute to this target.
Support for Docker has been removed for Kubernetes 1.24+. See https://kubernetes.io/blog/2020/12/02/dockershim-faq

Other changes of note

Full change list since 1.23.0 release

1.24.0-alpha.1 to 1.24.0-alpha.2

Update release notes and minimum k8s version @hakman #12929
kops auth-plugin: need to clear any existing password / key @justinsb #12921
Add integration test for k8s 1.24 @olemarkus #12930
Only shellcheck files @olemarkus #12931
Do not set insecure-port as of k8s 1.20 @olemarkus #12926
tests: Improve logging on test failure @justinsb #12933
nodeup: store the CloudProvider in the context @justinsb #12923
bazel: always build with pure (CGO_ENABLED=0) @justinsb #12934
nodeup: print more info on hash mismatches @justinsb #12935
PKI library: Add initial support for EC keys @justinsb #12936
Recognize debian bullseye as having "broken" resolv.conf @justinsb #12937
Remove code for now-unsupported Kubernetes 1.18 @johngmyers #12939
Add missing k8s 1.18 relnote @johngmyers #12938
Remove obsolete, redundant secrets.md @johngmyers #12942
Drop support for Weave as of k8s 1.23 @johngmyers #12941
Remove support for Aliyun/Alibaba Cloud @johngmyers #12944
Document CoreDNS configuration settings @recollir #12914
Update name of kubernetes-ca keypair in documentation @johngmyers #12943
Revert "Recognize debian bullseye as having "broken" resolv.conf" @olemarkus #12947
Set the default LT version to the new LT version @olemarkus #12932
Make service topology for cilium configurable @olemarkus #12918
gce: ServiceAccount task @justinsb #12950
Update Calico and Canal to v3.21.2 @hakman #12951
Update Go to v1.17.5 @hakman #12954
Skip IPv6 LB test in the k/s e2e @hakman #12953
GCE: Task for StorageBucket IAM @justinsb #12958
GCE: Project IAM Binding task @justinsb #12959
add verify-golangci-lint.sh script @rlankfo #12892
Hubble relay should not tolerate anything @olemarkus #12963
Do not explicitly skip Dashboard tests @hakman #12962
Do not skip NodePort tests for Calico @hakman #12960
Remove verify-staticcheck @rifelpet #12965
wait for instances to drain from classic LB @heybronson #12902
Support Karpenter @olemarkus #12906
Update containerd to v1.6.0-beta.4 @hakman #12968
Update controller-runtime to v0.11.0 @hakman #12967
Add missing permissions @olemarkus #12977
Do not skip HPA tests @hakman #12972
Do not skip RuntimeClass tests @hakman #12974
gce: Use ServiceAccount task when building model @justinsb #12978
Quote values and remove limits in karpenter provisioners @olemarkus #12979
Promote alpha with December releases @olemarkus #12984
gce: map multiple serviceaccounts @justinsb,@hakman #12982
Defend against nil containerd @justinsb #12990
Remove unused TemplateResource interface @justinsb #12989
Avoid double-encoding templates @justinsb #12991
Refactor nodeup script to avoid action-at-a-distance @justinsb #12993
gce: use per InstanceGroup serviceaccounts @justinsb #12988
dep: update github.com/pkg/sftp @justinsb #12996
Create helper functions for parsing public keys @justinsb #12999
Use terraform literals in GCP service account references @rifelpet #12995
kops-controller: use controller-runtime manager @justinsb #12997
gce: clean up networking objects by reference @justinsb #12987
componentconfig: expose advertise-address flag for kube-apiserver @justinsb #12998
Do not allow docker on k8s 1.24+ @olemarkus #12927
Ignore images hosted in private ECR repositories as containerd cannot pull these @olemarkus #13000
Skip RuntimeClass tests for older Kubernetes versions @hakman #13003
Various nill pointer fixes for karpenter @olemarkus #12973
Set Resource Based Naming on managed subnets @johngmyers #12864
Add kubetest2-kops flags for overriding instance group fields @rifelpet #13005
Support creating dualstack internal NLBs @johngmyers #13006
Skip SCTP check for all versions of k8s 1.23/1.24 @olemarkus #13008
Use spread constraints rather than affinity to spread pods @olemarkus #12961
Bump karpenter to 0.5.3 and RBN support @olemarkus #13002
Validate IGs more strictly after defaults have applied @olemarkus #12660
Karpenter template fix @olemarkus #13009
staticcheck cleanup: fixup nodeup/pkg/model @justinsb #13013
nodeup bash script: use explicit return code @justinsb #13012
Prevent creation of unsupported etcd clusters @olemarkus #13011
Create cgroups for kube and runtime if configured @olemarkus #12917
Do not install ClusterRole and binding used by in-tree volume provider if CSI is used @olemarkus #13010
kubetest2 - Use the same binary path and env when fetching IGs @rifelpet #13018
Use fi.Keyset instead of passing tasks around @justinsb #12992
add instance connection draining for NLBs @heybronson #12966
Use kubelet --non-masquerade-cidr only for Docker with kubenet @hakman #13007
Fix dangling ENIs from AWS VPC CNI @olemarkus #13021
Update k8s dependencies to v1.23.1 @hakman #13022
Improve HA for various addons @olemarkus #13027
Add a CLI flag for creating one karpenter-managed IG for worker nodes instead of ASG-managed ones @olemarkus #12975
Allow IPv6-only subnets @johngmyers #13026
Support specifying instance requirements per IG @olemarkus #13019
Remove TerraformJSON feature flag @rifelpet #13029
LBC has to run on the control plane, so set replicas accordingly @olemarkus #13033
Fix various typos related to karpenter @olemarkus #13035
Kube components log to stdout @olemarkus #13038
Identify pending instances @olemarkus #13040
Add managed-by label to static kube-proxy pods @olemarkus #13039
Prefix karpenter logging-config name @olemarkus #13037
gce: don't set per-IG permissions when using shared account @justinsb #13043
Add documentation on karpenter @olemarkus #13036
external CCM for GCE @jiahuif #13017
Migrate to GCE CCM in k8s 1.24 @johngmyers #13045
Fix OpenStack SecurityGroupRule/LB When CIDR is IPv6 @iGene #13032
update deps @zetaab #13047
Bump Cluster Autoscaler and update manifest @olemarkus #13050
Use instance requirements with Karpenter @olemarkus #13031
force update dependencies @zetaab #13055
Enhance AddHostPathMapping to support a fluent style @justinsb #13062
addons: support for kopeio-networking addon @justinsb #12727
Use latest GCP CCM for k8s 1.24 @johngmyers #13066
Add action for automatically tagging releases @johngmyers #12805
Bump external-snapshotted to v5.0.0 @olemarkus #13067
Release 1.24.0-alpha.2 @johngmyers #13069

1.24.0-alpha.2 to 1.24.0-alpha.3

Release notes for 1.24.0-alpha.2 @johngmyers #13070
Update release process for automatic tagging @johngmyers #13075
Remove temporary restrictions on automatically tagging releases @johngmyers #13071
add flatcar note related to additionalUserData @shubhindia #13061
Drain OpenStack loadbalancers @zetaab #12983
Extend terraform support for IPv6 @rifelpet #13028
Update containerd to v1.6.0-beta.5 @hakman #13084
Release notes for 1.22.3 @johngmyers #13085
Spotinst: Update spotinst/ocean-controller to v1.0.81 @liranp #13086
Support price and priority cluster-autoscaler expanders @danports #13081
Update containerd to v1.6.0-rc.0 @hakman #13098
decrease the openstack monitoring default timeout @zetaab #13097
Don't try to add node name to instances without node object @olemarkus #13106
fix ipv4+ipv6 sec groups/listeners in OpenStack @zetaab #13093
Do not create an IAM role for dns-controller on gossip clusters @olemarkus #13110
Add ipv6 to relnotes @olemarkus #13088
Use IPv6-only subnets for worker nodes in private IPv6 topology @johngmyers #13030
Remove networking flags as of k8s 1.24 @olemarkus #13120
Create helper function for ec2 create/tag-on-create IAM permissions @olemarkus #13104
Add DescribeRegions to nodeup privs @olemarkus #13114
Remove featureflag for creating IPv6 clusters @hakman #12788
Preload channel versions from namespaces @olemarkus #13049
Don't set unsupported configs by default @olemarkus #13111
Update pause image to v3.6 @hakman #13125
Clean up kubelet networking flags for dockershim @hakman #13128
January bump of channels @olemarkus #13130
expose external ccm metrics for OpenStack @zetaab #13131
Update to aws-sdk-go to v1.42.37 @jinhong- #13132
Fix recommended kops versions in channels @olemarkus #13134
Tag on create for remaining CCM privileges @olemarkus #12911
Bump metrics-server to 0.6.0 and enable HA mode @olemarkus #13135
OpenStack - Add loadbalancer pool monitor to API LB @zetaab #13096
Bump CCM images @olemarkus #13143
Bump karpenter to 0.5.6 @olemarkus #13151
Promote alpha AMIs to stable @yurrriq #13152
Bump 1.23 version in alpha channel @olemarkus #13153
Add missing v prefix to default upgrade test version @olemarkus #13155
Bump cert-manager and related godep to 1.6.2 @olemarkus #13154
add node-drain-timeout flag to rolling-update @heybronson #13103
Bump etcd-manager to v3.0.20220128 @olemarkus #13158
Replace deprecated aws.BackgroundContext with context.Background @justinsb #13162
Fix nil pointer when IAM not populated @justinsb #13167
JWKS / IRSA: Expose public ACLs to terraform @justinsb #13166
[DigitalOcean] update ccm version to 0.1.36 @srikiz #13175
Bump Ubuntu AMI in alpha @olemarkus #13177
Use etcd-manager pre-release until final release has been cut @olemarkus #13183
Bump karpenter to 0.6.0 @olemarkus #13185
More descriptive error message when public key file can't be opened @nckturner #13186
update GCE default images @zetaab #13181
Fix etcd-manager for ipv6 @olemarkus #13191
Update Calico and Canal to v3.21.4 @hakman #13189
Update to etcd-manager v3.0.20220203 @justinsb #13196
Pull k8s-custom-iptables from k8s.gcr.io @justinsb #13194
Add support for AB tests starting out with released kops version @olemarkus #13174
Update containerd to v1.6.0-rc.2 @hakman #13198
tests: ensure that we use ACLs with memfs @justinsb #13165
Karpenter fixes @olemarkus #13207
Always enable Leader Election for cloud-controller-manager @jiahuif #13187
Use short commit sha for default stage location instead of git-describe @olemarkus #13208
use 1.23.1 ccm for openstack @zetaab #13136
Document download of test versions @olemarkus #13209
Remove snapshot controller dependency on ebs csi driver @olemarkus #13213
fix KCM LogLevel setting not honored @jiahuif #13218
Fix CSI migration feature gates @olemarkus #13203
CCM: use flagbuilder instead of manually building argv @jiahuif #13219
Update containerd to v1.6.0-rc.3 @hakman #13224
Promote alpha to stable @MoShitrit #13227
always enable Leader Election for openstack CCM @jiahuif #13220
Update aws node termination handler to 1.14.0 @ryan-dyer-sp,@ryan-dyer #13092
[Issue-12293] Fix json output to keep it consistent for single or multiple objects @srikiz #13188
Fix irsa for k8s < 1.20 @olemarkus #13212
enable pruning for CCM @jiahuif #13235
Add support for graceful node shutdown @olemarkus #12994
allow specify GCP project via env. @jiahuif #13237
KCM should not run with leader migraton when aws ccm is enabled @olemarkus #13241
Do not enable graceful shutdown if k8s version < 1.21 @olemarkus #13242
Update metrics-server e2e test for 0.6.0 @olemarkus #13243
Install runc from opencontainers/runc @hakman #13240
Fix nilpointer when graceful shutdown is not configured @olemarkus #13246
Install contained from the release package @hakman #13248
CCM: allow setting Controllers for cloudControllerManagerConfig @jiahuif #13252
CCM: add livenessProbe for GCP CCM @jiahuif #13253
E2E HA Upgrade/Rollback for Leader Migration @jiahuif #13251
Bump AWS CNI to 1.10.2 @MoShitrit #13228
Update supported distros for IPv6 @hakman #13256
Karpenter on kOps will now use approperiate max pods @olemarkus #13178
Allow PrefixList for sshAccess and kubernetesApiAccess @hierynomus #13113
service account workaround for gce @jiahuif #13261
GCP API health checks @zetaab #13199
Update containerd to v1.6.0 @hakman #13262
re-organize Leader Migration test with exec tester @jiahuif #13265
Update LBC to 2.4.0 @olemarkus #13267
Enable RBN with AWS CCM 1.22.0-alpha.1 @johngmyers #13268
Disable some flags in kube-apiserver when logging-format is not text @h3poteto #13264
kops: Leader Migration testing: run with pure kubetest2 @jiahuif #13276
Bump k8s versions in alpha with Feb 2022 releases @MoShitrit #13275
Validate taints in IG spec @olemarkus #13266
test: use T.TempDir to create temporary test directory @Juneezee #13283
Do not create a cert-manager namespace @olemarkus #13284
Add missing permissions to aws lbc for irsa @olemarkus #13280
[DigitalOcean] Implement new VPC if network-cidr flag is specified @srikiz #13060
Use current tree in presubmit upgrade jobs if version B is latest @olemarkus #13290
Release notes for 1.22.4 @justinsb #13294
alpha channel: recommend kOps 1.22.4 @justinsb #13296
docs: add hubble ui helm chart deployment @eddycharly #13299
cleanup GCP Cluster Service Accounts @zetaab #13201
docs for release process shouldn't assume remotes @justinsb #13295
Release notes for 1.23.0-beta.2 @hakman #13303
Add support to install EKS Pod Identity Webhook @h3poteto,@olemarkus #13176
Update kubetest2 deps @olemarkus #13314
use own function to define CSI image version @zetaab #13311
Add support for ed25519 keys in AWS @aclevername #13304
Bump AWS SDK to v1.43.11 @olemarkus #13322
Make cloudProvider a struct in v1alpha3 API @johngmyers #13059
Update containerd to v1.6.1 @hakman #13325
Fix GCE service account creation @zetaab #13310
Use proper image and add health check @olemarkus #13328
Update stable and alpha channels @olemarkus #13334
Release notes for 1.21.5 @hakman #13336
Add e2e for pod identity webhook @olemarkus #13335
Add webhook notes + some docs changes @olemarkus #13338
Only delete node object on GCE @olemarkus #13289
Release notes for 1.23.0 @hakman #13340
Bump AWS CCM to 1.22.0-alpha.2 @olemarkus #13342
Bump CCM 1.22 image. Use the 1.23 image for 1.24 due to latest being broken @olemarkus #13357
Update channels @hakman #13356
Recommend enabling IRSA for new clusters @olemarkus #12976
Post 1.23.0 release doc updates @johngmyers #13359
Add user to container securityContext and remove command @olemarkus #13343
[Digital Ocean] e2e tests - Fix seeding for generating random zones @srikiz #13362
wait for all targetGroups to drain @heybronson #13363
Support GPU in OpenStack @zetaab #13330
Add missing permissions to aws lbc for IP targeting @olemarkus #13369
If kubetest2 fails cluster validation, we run down before exiting @olemarkus #13373
If image is empty, have kops upgrade fill it in @olemarkus #13374
Update channels @hakman #13379
Update HPA docs @ddelange #13367
Clean up nodeup targets @olemarkus #13370
Upgrade aws-iam-authenticator to v0.5.5 @glebiller #13381
Add protocol explicitly to services @olemarkus #13383
Allow duplicate taint keys @olemarkus #13366
[Digital Ocean] Remove sfo2 region from the list of supported DO regions @srikiz #13382
Fix long role names @olemarkus #13364
Migrate to registry.k8s.io @hakman #13380
Remove oss-upload target since aliyun support has been removed @olemarkus #13389
dev: create scripts to make it easier to run e2e tests @justinsb #13161
Remove pr target @olemarkus #13392

37 KiB Raw Blame History