## Release notes for kops 1.12 series

# Significant changes

* kops 1.12 enables etcd-manager by default. For kubernetes 1.12 (and later) we
  default to etcd3. We also enable TLS for etcd communications when using
  etcd-manager. **The upgrade is therefore disruptive to the masters.** More information is in the [etcd migration
  documentation](../etcd3-migration.md).
  This documentation is useful even if you are already using etcd3 with TLS.
  
* Components are no longer allowed to interact with etcd directly. Calico will
  be switched to use CRDs instead of directly with etcd. This is a disruptive
  upgrade, please read the calico notes in the [etcd migration
  documentation](../etcd3-migration.md)

# Required Actions

* Please back-up important data before upgrading, as the [etcd2 to etcd3
  migration](../etcd3-migration.md)
  is higher risk than most upgrades. **The upgrade is  disruptive to the masters, see notes above.**
* Note that **the upgrade for Calico users is disruptive**, because it requires
  switching from direct-etcd-storage to CRD backed storage.

# Full change list since 1.11.0 release

# 1.11.0 to 1.12.0-alpha.1

* machine-type generator: Warn if instance type not in ENI map [@justinsb](https://github.com/justinsb) [#6118](https://github.com/kubernetes/kops/pull/6118)
* Include name of unhealthy component in validation error [@justinsb](https://github.com/justinsb) [#6122](https://github.com/kubernetes/kops/pull/6122)
* Bump alpha channel kubernetes versions [@justinsb](https://github.com/justinsb) [#6123](https://github.com/kubernetes/kops/pull/6123)
* Add missing locking to awsmock LaunchConfigurations [@justinsb](https://github.com/justinsb) [#6124](https://github.com/kubernetes/kops/pull/6124)
* Add a1 and c5n instance types [@justinsb](https://github.com/justinsb) [#6117](https://github.com/kubernetes/kops/pull/6117)
* Simplify makefile for update-machine-types [@justinsb](https://github.com/justinsb) [#6121](https://github.com/kubernetes/kops/pull/6121)
* Update docs, removing brew --devel [@mikesplain](https://github.com/mikesplain) [#6125](https://github.com/kubernetes/kops/pull/6125)
* machine-types: remove duplicate dedup [@justinsb](https://github.com/justinsb) [#6127](https://github.com/kubernetes/kops/pull/6127)
* Update amazon cni to 1.3.0 [@mikesplain](https://github.com/mikesplain) [#6128](https://github.com/kubernetes/kops/pull/6128)
* Enable HPA tolerance configuration [@rlees85](https://github.com/rlees85) [#6130](https://github.com/kubernetes/kops/pull/6130)
* Update addons dashboard version [@jeefy](https://github.com/jeefy) [#6136](https://github.com/kubernetes/kops/pull/6136)
* Spotinst: Bump controller image [@liranp](https://github.com/liranp) [#6129](https://github.com/kubernetes/kops/pull/6129)
* Add cni to usage network option for kops create cluster [@nak3](https://github.com/nak3) [#6139](https://github.com/kubernetes/kops/pull/6139)
* Workspace updates for bazel / fix tests [@mikesplain](https://github.com/mikesplain) [#6144](https://github.com/kubernetes/kops/pull/6144)
* Promote alpha channels to stable [@mikesplain](https://github.com/mikesplain) [#6146](https://github.com/kubernetes/kops/pull/6146)
* Add GCE europe-north1-{a,b,c} [@eetujalonen](https://github.com/eetujalonen) [#6152](https://github.com/kubernetes/kops/pull/6152)
* Add self to security contacts [@mikesplain](https://github.com/mikesplain) [#6147](https://github.com/kubernetes/kops/pull/6147)
* Fix missed stable channel upgrade path [@mikesplain](https://github.com/mikesplain) [#6158](https://github.com/kubernetes/kops/pull/6158)
* Fix Calico upgrade job to use the correct version [@tmjd](https://github.com/tmjd) [#6156](https://github.com/kubernetes/kops/pull/6156)
* Fix for when node and master use the same SG. [@rdrgmnzs](https://github.com/rdrgmnzs) [#6175](https://github.com/kubernetes/kops/pull/6175)
* Add experimental and metrics flags for docker [@rbtcollins](https://github.com/rbtcollins) [#6171](https://github.com/kubernetes/kops/pull/6171)
* Add y flag for upgrade command for consistency [@mikesplain](https://github.com/mikesplain) [#6177](https://github.com/kubernetes/kops/pull/6177)
* Add-ons spec example is missing "manifest". [@qlikcoe](https://github.com/qlikcoe) [#6170](https://github.com/kubernetes/kops/pull/6170)
* ExperimentalAllowedUnsafeSysctls has moved to AllowedUnsafeSysctls in k8s 1.11 [@rdrgmnzs](https://github.com/rdrgmnzs) [#6179](https://github.com/kubernetes/kops/pull/6179)
* Let a user specify the validation timeout when rotating a cluster. [@rdrgmnzs](https://github.com/rdrgmnzs) [#6185](https://github.com/kubernetes/kops/pull/6185)
* fix(docs): fix the compatibility matrics on hpa.md [@Cryptophobia](https://github.com/Cryptophobia) [#6193](https://github.com/kubernetes/kops/pull/6193)
* bump prometheus-operator version and deploy file [@zouyee](https://github.com/zouyee) [#6196](https://github.com/kubernetes/kops/pull/6196)
* update heapster version and mark it retired [@zouyee](https://github.com/zouyee) [#6195](https://github.com/kubernetes/kops/pull/6195)
* Add Docker 18.06.1 for CentOS and RHEL 7 [@bcorijn](https://github.com/bcorijn) [#6202](https://github.com/kubernetes/kops/pull/6202)
* Print --name with `kops update cluster` [@joshbranham](https://github.com/joshbranham) [#6208](https://github.com/kubernetes/kops/pull/6208)
* Add --post-drain-delay to rolling-update cluster command [@rifelpet](https://github.com/rifelpet) [#6211](https://github.com/kubernetes/kops/pull/6211)
* Adding kubernetes-dashboard v1.10.1 deployment to kops addons [@schweizerbolzonello](https://github.com/schweizerbolzonello) [#6224](https://github.com/kubernetes/kops/pull/6224)
* Consider pending pods to be a validation failure [@justinsb](https://github.com/justinsb) [#6231](https://github.com/kubernetes/kops/pull/6231)
* Adding support for the new Stockholm region [@liranp](https://github.com/liranp) [#6212](https://github.com/kubernetes/kops/pull/6212)
* Document how to update an existing vendored dependency [@justinsb](https://github.com/justinsb) [#6238](https://github.com/kubernetes/kops/pull/6238)
* Update to k8s 1.12 libraries [@rdrgmnzs](https://github.com/rdrgmnzs) [#5932](https://github.com/kubernetes/kops/pull/5932)
* Bump channels and bump alpha to latest [@mikesplain](https://github.com/mikesplain) [#6239](https://github.com/kubernetes/kops/pull/6239)
* Automagically use curl instead of wget if that's what's available [@eherot](https://github.com/eherot) [#6090](https://github.com/kubernetes/kops/pull/6090)
* cloudmock: replace unimplemented methods with interface embedding [@justinsb](https://github.com/justinsb) [#6243](https://github.com/kubernetes/kops/pull/6243)
* bazel: cleanup gobindata generation [@justinsb](https://github.com/justinsb) [#6235](https://github.com/kubernetes/kops/pull/6235)
* Update apimachinery for k8s 1.12 [@justinsb](https://github.com/justinsb) [#6245](https://github.com/kubernetes/kops/pull/6245)
* Bulk spelling fixes [@justinsb](https://github.com/justinsb) [#6242](https://github.com/kubernetes/kops/pull/6242)
* Don't panic when an etcd cluster is added [@justinsb](https://github.com/justinsb) [#6180](https://github.com/kubernetes/kops/pull/6180)
* Update aws-sdk-go to 1.16.9 [@justinsb](https://github.com/justinsb) [#6237](https://github.com/kubernetes/kops/pull/6237)
* Add p3dn.24xlarge [@mikesplain](https://github.com/mikesplain) [#6253](https://github.com/kubernetes/kops/pull/6253)
* Rationalize deserialiation code [@justinsb](https://github.com/justinsb) [#6259](https://github.com/kubernetes/kops/pull/6259)
* Always log when a retry loop fails [@justinsb](https://github.com/justinsb) [#6260](https://github.com/kubernetes/kops/pull/6260)
* Update compatibility for v1.11.0 [@mikesplain](https://github.com/mikesplain) [#6258](https://github.com/kubernetes/kops/pull/6258)
*  AWS SDK v1.16.11 [@gambol99](https://github.com/gambol99) [#6276](https://github.com/kubernetes/kops/pull/6276)
* nodeup: include underlying error in error message [@andrestc](https://github.com/andrestc) [#6279](https://github.com/kubernetes/kops/pull/6279)
* release process: add the relnotes command [@justinsb](https://github.com/justinsb) [#6269](https://github.com/kubernetes/kops/pull/6269)
* Fix missed error check in hasPlaceHolderIP [@justinsb](https://github.com/justinsb) [#6272](https://github.com/kubernetes/kops/pull/6272)
* Create dev-upload tasks, for a faster upload during dev builds [@justinsb](https://github.com/justinsb) [#6233](https://github.com/kubernetes/kops/pull/6233)
* Update recommended kubernetes version [@justinsb](https://github.com/justinsb) [#6271](https://github.com/kubernetes/kops/pull/6271)
* Release notes for 1.11 [@justinsb](https://github.com/justinsb) [#6270](https://github.com/kubernetes/kops/pull/6270)
* fixed the sentence mistake [@abhijitio](https://github.com/abhijitio) [#6281](https://github.com/kubernetes/kops/pull/6281)
* update calico version to version 3.4.0 [@chrisz100](https://github.com/chrisz100) [#6263](https://github.com/kubernetes/kops/pull/6263)
* Remove duplicate Deployment for prometheus-operator [@Smirl](https://github.com/Smirl) [#6265](https://github.com/kubernetes/kops/pull/6265)
* Update aws-china.md [@qqshfox](https://github.com/qqshfox) [#6262](https://github.com/kubernetes/kops/pull/6262)
* Recognize 2019 as a year [@justinsb](https://github.com/justinsb) [#6288](https://github.com/kubernetes/kops/pull/6288)
* Change jessie to stretch [@abhijitio](https://github.com/abhijitio) [#6293](https://github.com/kubernetes/kops/pull/6293)
* Included type in SSL certificate documentation [@walkafwalka](https://github.com/walkafwalka) [#6289](https://github.com/kubernetes/kops/pull/6289)
* Update distroless [@justinsb](https://github.com/justinsb) [#6287](https://github.com/kubernetes/kops/pull/6287)
* Promote alpha kubernetes versions to stable [@justinsb](https://github.com/justinsb) [#6298](https://github.com/kubernetes/kops/pull/6298)
* Create prow-postsubmit target for release candidates [@justinsb](https://github.com/justinsb) [#6299](https://github.com/kubernetes/kops/pull/6299)
* Include windows build in distribution [@justinsb](https://github.com/justinsb) [#6300](https://github.com/kubernetes/kops/pull/6300)
* Fix kubelet api admin [@gambol99](https://github.com/gambol99) [#6312](https://github.com/kubernetes/kops/pull/6312)
* GCE terraform: map source tags in firewallrule [@justinsb](https://github.com/justinsb) [#6295](https://github.com/kubernetes/kops/pull/6295)
* GCE terraform: support labels [@justinsb](https://github.com/justinsb) [#6296](https://github.com/kubernetes/kops/pull/6296)
* Add extra privilege to prometheus-k8s ClusterRole  [#6305](https://github.com/kubernetes/kops/pull/6305)
* Kubelet API RBAC Manifest [@gambol99](https://github.com/gambol99) [#6317](https://github.com/kubernetes/kops/pull/6317)
* Upgrading coredns version to 1.3.0 [@harshal-shah](https://github.com/harshal-shah) [#6326](https://github.com/kubernetes/kops/pull/6326)
* Release 1.12.0-alpha.1 [@justinsb](https://github.com/justinsb) [#6257](https://github.com/kubernetes/kops/pull/6257)
* Retry Logging [@gambol99](https://github.com/gambol99) [#6327](https://github.com/kubernetes/kops/pull/6327)
* Fix prow-postsubmit by copying prebuilt archive in bazel [@justinsb](https://github.com/justinsb) [#6328](https://github.com/kubernetes/kops/pull/6328)
* Remove Initializers from default admission plugins for 1.12+ [@liggitt](https://github.com/liggitt) [#6350](https://github.com/kubernetes/kops/pull/6350)
* include docker 18.06.1 missed dependency [@nareshku](https://github.com/nareshku) [#6338](https://github.com/kubernetes/kops/pull/6338)
* Fix alternative AWS partitions in custom instance profiles [@rifelpet](https://github.com/rifelpet) [#6226](https://github.com/kubernetes/kops/pull/6226)
* Add doc regarding upgrading to CoreDNS [@joshbranham](https://github.com/joshbranham) [#6344](https://github.com/kubernetes/kops/pull/6344)
* AWS: Enable ICMP Type 3 Code 4 for API server ELBs [@davidarcher](https://github.com/davidarcher) [#6297](https://github.com/kubernetes/kops/pull/6297)
* Additional Storage & Volume Mounting [@gambol99](https://github.com/gambol99) [#6066](https://github.com/kubernetes/kops/pull/6066)
* kOps for Openstack [@jrperritt](https://github.com/jrperritt),[@drekle](https://github.com/drekle),[@wozniakjan](https://github.com/wozniakjan),[@marsavela](https://github.com/marsavela) [#6351](https://github.com/kubernetes/kops/pull/6351)
* Update go version to 1.10.8 [@justinsb](https://github.com/justinsb) [#6401](https://github.com/kubernetes/kops/pull/6401)
* Suffix openstack subnet name with cluster name [@wozniakjan](https://github.com/wozniakjan) [#6380](https://github.com/kubernetes/kops/pull/6380)
* Update upgrade.md [@ms4720](https://github.com/ms4720) [#6396](https://github.com/kubernetes/kops/pull/6396)
* minor grammar improvements to kops terraform docs [@discdiver](https://github.com/discdiver) [#6301](https://github.com/kubernetes/kops/pull/6301)
* Docs: Drop last DrainAndValidateRollingUpdate note [@meeee](https://github.com/meeee) [#6374](https://github.com/kubernetes/kops/pull/6374)
* Allow users to set kubelet cpu-cfs-quota and cpu-cfs-quota-period flags [@wndhydrnt](https://github.com/wndhydrnt) [#6375](https://github.com/kubernetes/kops/pull/6375)
* implement etcd status for openstack [@zetaab](https://github.com/zetaab) [#6381](https://github.com/kubernetes/kops/pull/6381)
* remove using deviceowner when filtering existing routerinterfaces [@zetaab](https://github.com/zetaab) [#6382](https://github.com/kubernetes/kops/pull/6382)
* ignore openstack managed volume tags [@zetaab](https://github.com/zetaab) [#6383](https://github.com/kubernetes/kops/pull/6383)
* kops version: Add --short flag, use it to get version in scripts [@justinsb](https://github.com/justinsb) [#6232](https://github.com/kubernetes/kops/pull/6232)
* find sshkey resource when updating cluster [@zetaab](https://github.com/zetaab) [#6384](https://github.com/kubernetes/kops/pull/6384)
* implement GetCloudGroups for openstack [@zetaab](https://github.com/zetaab) [#6386](https://github.com/kubernetes/kops/pull/6386)
* minor fixes to openstack [@zetaab](https://github.com/zetaab) [#6387](https://github.com/kubernetes/kops/pull/6387)
* fix openstack lb pool member logic [@zetaab](https://github.com/zetaab) [#6388](https://github.com/kubernetes/kops/pull/6388)
* Support "egress: External" to avoid configuring networking [@justinsb](https://github.com/justinsb),[@cassandracomar](https://github.com/cassandracomar),[@moustafab](https://github.com/moustafab) [#6218](https://github.com/kubernetes/kops/pull/6218)
* Bump alpha channels [@mikesplain](https://github.com/mikesplain) [#6405](https://github.com/kubernetes/kops/pull/6405)
* Update bazel rules [@mikesplain](https://github.com/mikesplain) [#6406](https://github.com/kubernetes/kops/pull/6406)
* implement delete cluster for openstack [@zetaab](https://github.com/zetaab) [#6385](https://github.com/kubernetes/kops/pull/6385)
* Openstack Floating IP Deletion [@drekle](https://github.com/drekle) [#6425](https://github.com/kubernetes/kops/pull/6425)
* update openstack documentation [@zetaab](https://github.com/zetaab) [#6423](https://github.com/kubernetes/kops/pull/6423)
* Updated OWNERS file to include link to docs [@rlenferink](https://github.com/rlenferink) [#6450](https://github.com/kubernetes/kops/pull/6450)
* [jjo] add docker-ce 18.06.2 for CVE-2019-5736 [@jjo](https://github.com/jjo) [#6460](https://github.com/kubernetes/kops/pull/6460)
* Add permission for CreateTag on ENI to amazon-vpc-cni-k8s [@nak3](https://github.com/nak3) [#6389](https://github.com/kubernetes/kops/pull/6389)
* Document etcd3 migration process [@justinsb](https://github.com/justinsb) [#6408](https://github.com/kubernetes/kops/pull/6408)
* Normalize etcd cluster provider names [@justinsb](https://github.com/justinsb) [#6410](https://github.com/kubernetes/kops/pull/6410)
* Support etcd-manager v3, suitable for backporting [@justinsb](https://github.com/justinsb) [#6411](https://github.com/kubernetes/kops/pull/6411)
* Openstack loadbalancers erronous modification requests [@drekle](https://github.com/drekle) [#6413](https://github.com/kubernetes/kops/pull/6413)
* fix typos for addon doc [@fatsheep9146](https://github.com/fatsheep9146) [#6416](https://github.com/kubernetes/kops/pull/6416)
* upgrade calico to 2.6.12 to fix TTA-2018-001 [@mechpen](https://github.com/mechpen) [#6422](https://github.com/kubernetes/kops/pull/6422)
* Use the forward plugin instead of proxy plugin in CoreDNS [@rajansandeep](https://github.com/rajansandeep) [#6424](https://github.com/kubernetes/kops/pull/6424)
* Update bazel workspace [@mikesplain](https://github.com/mikesplain) [#6426](https://github.com/kubernetes/kops/pull/6426)
* Fix machine types and cleanup makefile [@mikesplain](https://github.com/mikesplain) [#6427](https://github.com/kubernetes/kops/pull/6427)
* Add jessie patch [@jjo](https://github.com/jjo),[@mikesplain](https://github.com/mikesplain) [#6461](https://github.com/kubernetes/kops/pull/6461)
* Allow NodeAuthorizer to speak via HTTP Proxy if configured [@KashifSaadat](https://github.com/KashifSaadat) [#6468](https://github.com/kubernetes/kops/pull/6468)
* Updated Canal manifest to v3.5.0 for k8s v1.12+ [@KashifSaadat](https://github.com/KashifSaadat) [#6469](https://github.com/kubernetes/kops/pull/6469)
* Update document for GPU support [@yujunz](https://github.com/yujunz) [#6246](https://github.com/kubernetes/kops/pull/6246)
* Fixing kops-4049 [@mmerrill3](https://github.com/mmerrill3) [#6210](https://github.com/kubernetes/kops/pull/6210)
* kube-apiserver: Add oidc-required-claim flag [@jeyglk](https://github.com/jeyglk) [#6453](https://github.com/kubernetes/kops/pull/6453)
* add OWNERS file to openstack spesific folders [@zetaab](https://github.com/zetaab) [#6367](https://github.com/kubernetes/kops/pull/6367)
* Update Loadbalancer Pools [@drekle](https://github.com/drekle) [#6433](https://github.com/kubernetes/kops/pull/6433)
* fix hostnames in kops openstack [@zetaab](https://github.com/zetaab) [#6442](https://github.com/kubernetes/kops/pull/6442)
* implement ig deletegroup for openstack [@zetaab](https://github.com/zetaab) [#6418](https://github.com/kubernetes/kops/pull/6418)
* Removing openstack credential file support [@drekle](https://github.com/drekle) [#6480](https://github.com/kubernetes/kops/pull/6480)
* fix error when updating/creating lb in openstack [@zetaab](https://github.com/zetaab) [#6431](https://github.com/kubernetes/kops/pull/6431)
* recheck floatingip after server is active [@zetaab](https://github.com/zetaab) [#6432](https://github.com/kubernetes/kops/pull/6432)
* Ability to scale down instancegroup in openstack [@zetaab](https://github.com/zetaab) [#6421](https://github.com/kubernetes/kops/pull/6421)
* expose DryRunTarget changes and deletions [@zetaab](https://github.com/zetaab) [#6415](https://github.com/kubernetes/kops/pull/6415)
* support both octavia and old lbaasv2 api in openstack [@zetaab](https://github.com/zetaab) [#6438](https://github.com/kubernetes/kops/pull/6438)
* Guess SSH usernames for RHEL & Centos in toolbox dump [@justinsb](https://github.com/justinsb) [#6487](https://github.com/kubernetes/kops/pull/6487)
* Choose docker version 18.06.2 for k8s >= 1.12 [@justinsb](https://github.com/justinsb) [#6488](https://github.com/kubernetes/kops/pull/6488)
* Install kubelet config for default centos user [@justinsb](https://github.com/justinsb) [#6489](https://github.com/kubernetes/kops/pull/6489)
* Update the CoreDNS manifest [@rajansandeep](https://github.com/rajansandeep) [#6485](https://github.com/kubernetes/kops/pull/6485)
* docs: improve the queries for finding RHEL/CentOS images [@justinsb](https://github.com/justinsb) [#6486](https://github.com/kubernetes/kops/pull/6486)
* Workaround for overlay2 vs rhel-family docker bug [@justinsb](https://github.com/justinsb) [#6491](https://github.com/kubernetes/kops/pull/6491)
* retry l3floatingip list in fresh cluster [@zetaab](https://github.com/zetaab) [#6497](https://github.com/kubernetes/kops/pull/6497)
* Update 1.12 addon manifests to use apps/v1, rbac v1 [@liggitt](https://github.com/liggitt) [#6397](https://github.com/kubernetes/kops/pull/6397)
* Fix package name & version for container-selinux [@justinsb](https://github.com/justinsb) [#6492](https://github.com/kubernetes/kops/pull/6492)
* AWS Mixed Instances Policy / Fleet [@gambol99](https://github.com/gambol99) [#6277](https://github.com/kubernetes/kops/pull/6277)
* Adding Comment [@gambol99](https://github.com/gambol99) [#6508](https://github.com/kubernetes/kops/pull/6508)
* Kube Proxy Metrics Option [@gambol99](https://github.com/gambol99) [#6513](https://github.com/kubernetes/kops/pull/6513)
* Sprig (Toolbox Templating) [@gambol99](https://github.com/gambol99) [#6515](https://github.com/kubernetes/kops/pull/6515)
* Etcd memory and cpu requests [@integrii](https://github.com/integrii) [#6313](https://github.com/kubernetes/kops/pull/6313)
* Map docker 18.06.3 [@justinsb](https://github.com/justinsb) [#6523](https://github.com/kubernetes/kops/pull/6523)
* Make docker 18.06.3 the default for k8s >= 1.12 [@justinsb](https://github.com/justinsb) [#6524](https://github.com/kubernetes/kops/pull/6524)
* Document strategy for cve_2019_5736 [@justinsb](https://github.com/justinsb) [#6522](https://github.com/kubernetes/kops/pull/6522)
* Try using chattr to mark docker-runc as immutable [@justinsb](https://github.com/justinsb) [#6506](https://github.com/kubernetes/kops/pull/6506)
* Simple mirror support [@justinsb](https://github.com/justinsb) [#6503](https://github.com/kubernetes/kops/pull/6503)
* Bump etcd-manager version to 3.0.20190224 [@justinsb](https://github.com/justinsb) [#6526](https://github.com/kubernetes/kops/pull/6526)
* update gophercloud vendor dependencies [@zetaab](https://github.com/zetaab) [#6478](https://github.com/kubernetes/kops/pull/6478)
* specify dns servers to openstack subnet [@zetaab](https://github.com/zetaab) [#6530](https://github.com/kubernetes/kops/pull/6530)
* possibility to specify floatingip subnet for resources in openstack [@zetaab](https://github.com/zetaab) [#6477](https://github.com/kubernetes/kops/pull/6477)
* Add Experimental Cluster Signing Duration flag [@pgdagenais](https://github.com/pgdagenais) [#6525](https://github.com/kubernetes/kops/pull/6525)
* set net.ipv4.ip_local_reserved_ports to the KubeAPIServer ServiceNodePortRange parameter on nodeup [@sp-joseluis-ledesma](https://github.com/sp-joseluis-ledesma) [#6343](https://github.com/kubernetes/kops/pull/6343)
* spread instances equally to all AZs [@zetaab](https://github.com/zetaab) [#6534](https://github.com/kubernetes/kops/pull/6534)
* update-machine-types: more metal instance types [@justinsb](https://github.com/justinsb) [#6551](https://github.com/kubernetes/kops/pull/6551)
* Add changelist for 1.11.1 [@justinsb](https://github.com/justinsb) [#6565](https://github.com/kubernetes/kops/pull/6565)
* Fix panic when using etcd-manager and resource requests are nil [@KashifSaadat](https://github.com/KashifSaadat) [#6563](https://github.com/kubernetes/kops/pull/6563)
* Promote Kubernetes 1.11.7 to stable [@olemarkus](https://github.com/olemarkus) [#6566](https://github.com/kubernetes/kops/pull/6566)
* Upgrade alpha to latest [@mikesplain](https://github.com/mikesplain) [#6568](https://github.com/kubernetes/kops/pull/6568)
* implement delete instance, this is needed in rolling-update [@zetaab](https://github.com/zetaab) [#6576](https://github.com/kubernetes/kops/pull/6576)
* Stop setting deprecated --allow-privileged Kubelet flag in 1.14 [@mtaufen](https://github.com/mtaufen) [#6340](https://github.com/kubernetes/kops/pull/6340)
* Openstack Security Group hardening [@drekle](https://github.com/drekle) [#6521](https://github.com/kubernetes/kops/pull/6521)
* Update embargo doc link in SECURITY_CONTACTS and change PST to PSC [@joelsmith](https://github.com/joelsmith) [#6601](https://github.com/kubernetes/kops/pull/6601)
* Instance LaunchConfig/Template Bug Fix [@gambol99](https://github.com/gambol99) [#6590](https://github.com/kubernetes/kops/pull/6590)
* add docker.insecureRegistries flag [@kimxogus](https://github.com/kimxogus) [#6586](https://github.com/kubernetes/kops/pull/6586)
* Add line breaks in example release cycle [@MMeent](https://github.com/MMeent) [#6591](https://github.com/kubernetes/kops/pull/6591)
* [jjo] Update Weave Net to version 2.5.1 [@jjo](https://github.com/jjo) [#6370](https://github.com/kubernetes/kops/pull/6370)
* Adding installation guidelines for Windows [@EchoDelta](https://github.com/EchoDelta) [#6594](https://github.com/kubernetes/kops/pull/6594)
* Remove confusing comma in README [@mattjmcnaughton](https://github.com/mattjmcnaughton) [#6607](https://github.com/kubernetes/kops/pull/6607)
* Add ServiceAccountKeyFile to KubeAPIServerConfig [@Smirl](https://github.com/Smirl) [#6578](https://github.com/kubernetes/kops/pull/6578)
* moving chrisz100 to approver level [@chrisz100](https://github.com/chrisz100) [#6434](https://github.com/kubernetes/kops/pull/6434)
* Fix dashboard yaml that returned 404 [@mausch](https://github.com/mausch) [#6479](https://github.com/kubernetes/kops/pull/6479)
* Replace Y / N Markings of Compatibility Matrix in readme with ✔ / ❌ [@compilenix](https://github.com/compilenix) [#6539](https://github.com/kubernetes/kops/pull/6539)
* Rename addon.yml to addon.yaml [@jsharpe](https://github.com/jsharpe) [#6323](https://github.com/kubernetes/kops/pull/6323)
* addons/cluster-autoscaler: Add jq installation for OSX environment [@iBluemind](https://github.com/iBluemind) [#6567](https://github.com/kubernetes/kops/pull/6567)
* Update docs on authentication [@flands](https://github.com/flands) [#6575](https://github.com/kubernetes/kops/pull/6575)
* Omit IP-in-IP protocols in Openstack CNI Rules [@marsavela](https://github.com/marsavela) [#6614](https://github.com/kubernetes/kops/pull/6614)
* External out-of-tree CloudControllerManager support for openstack [@zetaab](https://github.com/zetaab) [#6444](https://github.com/kubernetes/kops/pull/6444)
* Use EnsureTask for create static pod directory [@Smirl](https://github.com/Smirl) [#6616](https://github.com/kubernetes/kops/pull/6616)
* Fix documentation about targetGroupArn key [@phyrog](https://github.com/phyrog) [#6611](https://github.com/kubernetes/kops/pull/6611)
* Update rolling_update.md [@rj03hou](https://github.com/rj03hou),[@Pharb](https://github.com/Pharb) [#6247](https://github.com/kubernetes/kops/pull/6247)
* fix typo [@zqm19941101](https://github.com/zqm19941101) [#6017](https://github.com/kubernetes/kops/pull/6017)
* Correcly handle CRLF in the manifest [@gtrafimenkov](https://github.com/gtrafimenkov) [#6570](https://github.com/kubernetes/kops/pull/6570)
* Add support for Docker 18.09.3. [@tsuna](https://github.com/tsuna) [#6347](https://github.com/kubernetes/kops/pull/6347)
* Fix confusing k8s upgrade docs for Terraform users [@tspacek](https://github.com/tspacek),[@justinsb](https://github.com/justinsb) [#6275](https://github.com/kubernetes/kops/pull/6275)
* Added Audit Webhook config [@mbelangerupgrade](https://github.com/mbelangerupgrade),[@jpbelangerupgrade](https://github.com/jpbelangerupgrade) [#6361](https://github.com/kubernetes/kops/pull/6361)
* Spotinst: Avoid spurious changes [@liranp](https://github.com/liranp) [#6028](https://github.com/kubernetes/kops/pull/6028)
* Fix amazon-vpc-routed-eni yaml template [@tvi](https://github.com/tvi) [#6502](https://github.com/kubernetes/kops/pull/6502)
* Replace gcr.io URL with k8s.gcr.io vanity URL [@justinsb](https://github.com/justinsb) [#6623](https://github.com/kubernetes/kops/pull/6623)
* support gossip for AliCloud [@LilyFaFa](https://github.com/LilyFaFa) [#6319](https://github.com/kubernetes/kops/pull/6319)
* add natGateways tasks for ALICloud [@LilyFaFa](https://github.com/LilyFaFa) [#6402](https://github.com/kubernetes/kops/pull/6402)
* Fix some of the docker package names & versions [@justinsb](https://github.com/justinsb) [#6620](https://github.com/kubernetes/kops/pull/6620)
* Apply scope fix in #6502 to all manifest versions [@tvi](https://github.com/tvi),[@justinsb](https://github.com/justinsb) [#6622](https://github.com/kubernetes/kops/pull/6622)
* Add --kubeconfig flag to `kops export kubecfg` [@adamyy](https://github.com/adamyy) [#5955](https://github.com/kubernetes/kops/pull/5955)
* add support to set cluster spec.kubelet [@phedoreanu](https://github.com/phedoreanu) [#6619](https://github.com/kubernetes/kops/pull/6619)
* Upgrade bazel gazelle [@mikesplain](https://github.com/mikesplain) [#6609](https://github.com/kubernetes/kops/pull/6609)
* Fix typo [@justinsb](https://github.com/justinsb) [#6621](https://github.com/kubernetes/kops/pull/6621)
* Support g3s for gpu driver installation [@reverson](https://github.com/reverson) [#6538](https://github.com/kubernetes/kops/pull/6538)
* Fix docker-healthcheck to work around Docker bug. [@tsuna](https://github.com/tsuna) [#6448](https://github.com/kubernetes/kops/pull/6448)
* docs: create checklist for new kubernetes version [@justinsb](https://github.com/justinsb) [#5818](https://github.com/kubernetes/kops/pull/5818)
* Fix metrics server addon [@itskingori](https://github.com/itskingori) [#6201](https://github.com/kubernetes/kops/pull/6201)
* Always create /var/lib/kubelet, even in bootstrap mode [@justinsb](https://github.com/justinsb) [#5982](https://github.com/kubernetes/kops/pull/5982)
* Launch Template Feature Flag [@gambol99](https://github.com/gambol99) [#6512](https://github.com/kubernetes/kops/pull/6512)
* Remove docker-prestart hook [@stevenjm](https://github.com/stevenjm) [#6564](https://github.com/kubernetes/kops/pull/6564)
* kops 1.12 configuration for calico: use CRDs [@justinsb](https://github.com/justinsb) [#6358](https://github.com/kubernetes/kops/pull/6358)
* Quick Clean [@gambol99](https://github.com/gambol99) [#6634](https://github.com/kubernetes/kops/pull/6634)
* Sync data-types for webhook config with upstream [@justinsb](https://github.com/justinsb) [#6626](https://github.com/kubernetes/kops/pull/6626)
* Add manage security groups for loadbalancers [@zetaab](https://github.com/zetaab) [#6632](https://github.com/kubernetes/kops/pull/6632)
* Enable etcd-manager / etcd3 / etcd-tls in kops 1.12 [@justinsb](https://github.com/justinsb) [#6359](https://github.com/kubernetes/kops/pull/6359)
* Use EnsureTask for internal api route53 record [@Smirl](https://github.com/Smirl) [#6629](https://github.com/kubernetes/kops/pull/6629)
* Added reminder to publish conformance results in release process [@chrisz100](https://github.com/chrisz100) [#6640](https://github.com/kubernetes/kops/pull/6640)
* Update aws-china.md [@qqshfox](https://github.com/qqshfox) [#6643](https://github.com/kubernetes/kops/pull/6643)
* Openstack server name collisions [@drekle](https://github.com/drekle) [#6650](https://github.com/kubernetes/kops/pull/6650)
* tiny backslash arrangement [@sevenfourk](https://github.com/sevenfourk) [#6652](https://github.com/kubernetes/kops/pull/6652)
* Openstack environment escaping [@drekle](https://github.com/drekle) [#6657](https://github.com/kubernetes/kops/pull/6657)
* Update upgrade.md [@gamename](https://github.com/gamename) [#6654](https://github.com/kubernetes/kops/pull/6654)
* add ALI flags [@LilyFaFa](https://github.com/LilyFaFa) [#6628](https://github.com/kubernetes/kops/pull/6628)
* Override volume zone name [@zetaab](https://github.com/zetaab) [#6655](https://github.com/kubernetes/kops/pull/6655)
* Updated Flannel manifest to 0.11.0 [@gordonbondon](https://github.com/gordonbondon) [#6660](https://github.com/kubernetes/kops/pull/6660)
* Update flannel version in bootstrapchannelbuilder [@gordonbondon](https://github.com/gordonbondon),[@justinsb](https://github.com/justinsb) [#6663](https://github.com/kubernetes/kops/pull/6663)
* Add flags for TLS Cipher suites customization for API Server, Kubelet and Controller-Manager [@rochacon](https://github.com/rochacon) [#6470](https://github.com/kubernetes/kops/pull/6470)
* If using etcd-backup and TLS is enabled, pass relevant options [@KashifSaadat](https://github.com/KashifSaadat) [#6562](https://github.com/kubernetes/kops/pull/6562)
* Bump etcd-manager / etcd-backup to 3.0.20190325 [@justinsb](https://github.com/justinsb) [#6664](https://github.com/kubernetes/kops/pull/6664)
* 2048 - Add cloudLabels as tags to API ELB resource [@ryan-dyer](https://github.com/ryan-dyer) [#6646](https://github.com/kubernetes/kops/pull/6646)
* Bump K8s 1.11 to 1.11.9 in the alpha channel [@olemarkus](https://github.com/olemarkus) [#6665](https://github.com/kubernetes/kops/pull/6665)
* Upgrade rules go [@mikesplain](https://github.com/mikesplain) [#6667](https://github.com/kubernetes/kops/pull/6667)
* Fix a missing dep lock [@mikesplain](https://github.com/mikesplain) [#6668](https://github.com/kubernetes/kops/pull/6668)

# 1.12.0-alpha.1 to 1.12.0-alpha.2

* Support download protokube from mirror [@justinsb](https://github.com/justinsb) [#6673](https://github.com/kubernetes/kops/pull/6673)
* Promote alpha to stable and update alpha [@mikesplain](https://github.com/mikesplain) [#6669](https://github.com/kubernetes/kops/pull/6669)
* Upload protokube to github as part of release [@justinsb](https://github.com/justinsb) [#6674](https://github.com/kubernetes/kops/pull/6674)
* Use CNI 0.7.5 [@justinsb](https://github.com/justinsb) [#6671](https://github.com/kubernetes/kops/pull/6671)
* Put 1.12 into stable channel, for users of kops 1.12-alphas [@justinsb](https://github.com/justinsb) [#6672](https://github.com/kubernetes/kops/pull/6672)
* Support mirrors with restricted characters [@justinsb](https://github.com/justinsb) [#6675](https://github.com/kubernetes/kops/pull/6675)

# 1.12.0-alpha.2 to 1.12.0-alpha.3

* Fix Key error change Overrides to Override [@granular-ryanbonham](https://github.com/granular-ryanbonham) [#6691](https://github.com/kubernetes/kops/pull/6691)
* Add selector back to calico 1.12 deployment [@justinsb](https://github.com/justinsb) [#6682](https://github.com/kubernetes/kops/pull/6682)
* Update etcd-manager to 1.0.20190328 [@justinsb](https://github.com/justinsb) [#6695](https://github.com/kubernetes/kops/pull/6695)

# 1.12.0-alpha.3 to 1.12.0-beta.1

* Fix tagging and remove tagging elbs [@mikesplain](https://github.com/mikesplain) [#6705](https://github.com/kubernetes/kops/pull/6705)
* Add DNS Resource Settings [@granular-ryanbonham](https://github.com/granular-ryanbonham) [#6731](https://github.com/kubernetes/kops/pull/6731)
* Update instances types [@mikesplain](https://github.com/mikesplain) [#6733](https://github.com/kubernetes/kops/pull/6733)
* Update kube-dns 1.3.0 to 1.3.3 [@mikesplain](https://github.com/mikesplain) [#6734](https://github.com/kubernetes/kops/pull/6734)
* kube-dns-autoscaler: Add node watch to permissions [@justinsb](https://github.com/justinsb) [#6740](https://github.com/kubernetes/kops/pull/6740)
* Increase apiserver timeout to 45 seconds [@justinsb](https://github.com/justinsb) [#6743](https://github.com/kubernetes/kops/pull/6743)
* Fix issue #6700: User Data for launch templates & other terraform issues [@rdrgmnzs](https://github.com/rdrgmnzs) [#6732](https://github.com/kubernetes/kops/pull/6732)

# 1.12.0-beta.1 to 1.12.0-beta.2

* kube-dns: Update to 1.14.13 [@justinsb](https://github.com/justinsb) [#6741](https://github.com/kubernetes/kops/pull/6741)
* Launch Template use version number as well as name.[@granular-ryanbonham](https://github.com/granular-ryanbonham) [#6755](https://github.com/kubernetes/kops/pull/6755)
* use dynamic s3 prefix in addAmazonVPCCNIPermissions func [@bksteiny](https://github.com/bksteiny) [#6765](https://github.com/kubernetes/kops/pull/6765)

# 1.12.0-beta.2 to 1.12.0

* IAM Permission to Support Scaling from 0 with Lauch Templates [@granular-ryanbonham](https://github.com/granular-ryanbonham) [#6861](https://github.com/kubernetes/kops/pull/6861)
* Avoid concurrent write corruption to /etc/hosts [@justinsb](https://github.com/justinsb),[@granular-ryanbonham](https://github.com/granular-ryanbonham) [#6893](https://github.com/kubernetes/kops/pull/6893)
* Add i3en instance types [@mikesplain](https://github.com/mikesplain) [#6898](https://github.com/kubernetes/kops/pull/6898)
* Add t3a family [@mikesplain](https://github.com/mikesplain) [#6905](https://github.com/kubernetes/kops/pull/6905)
* Use existing SSHKeyName if no public key is created. [@rralcala](https://github.com/rralcala) [#6886](https://github.com/kubernetes/kops/pull/6886)
* bazel: fix distroless imports for latest bazel [@justinsb](https://github.com/justinsb) [#6910](https://github.com/kubernetes/kops/pull/6910)
* pkg/model: Fix dropped error [@alrs](https://github.com/alrs) [#6911](https://github.com/kubernetes/kops/pull/6911)
* Add ability to specify cpuRequest for API Server [@granular-ryanbonham](https://github.com/granular-ryanbonham) [#6706](https://github.com/kubernetes/kops/pull/6706)
* KubeAPIServer HTTP2 Stream Parameter [@gambol99](https://github.com/gambol99) [#6913](https://github.com/kubernetes/kops/pull/6913)
* Add support for AWS ap-east-1 region [@wxdao](https://github.com/wxdao) [#6835](https://github.com/kubernetes/kops/pull/6835)
* Add min-resync-period for Controller Manager [@maruina](https://github.com/maruina) [#6737](https://github.com/kubernetes/kops/pull/6737)
* Allow the AWS IAM Authenticator image name to be overridden [@rifelpet](https://github.com/rifelpet) [#6730](https://github.com/kubernetes/kops/pull/6730)
* Add cpu management policy config [@lynchc](https://github.com/lynchc) [#5961](https://github.com/kubernetes/kops/pull/5961)
* Carry Provisioned IOPS to Terraform and CloudFormation templates [@MathieuMailhos](https://github.com/mmailhos) [#6776](https://github.com/kubernetes/kops/pull/6776)
* update tolerations to openstack external cloud provider [@zetaab](https://github.com/zetaab) [#6821](https://github.com/kubernetes/kops/pull/6821)
* Fix typo in aws-iam-authenticator image field name [@rifelpet](https://github.com/rifelpet) [#6840](https://github.com/kubernetes/kops/pull/6840)
* add the registry-qps kubelet flag [@sp-joseluis-ledesma](https://github.com/sp-joseluis-ledesma) [#6357](https://github.com/kubernetes/kops/pull/6357)
* Deep-copy proto state to prevent concurrent modification [@justinsb](https://github.com/justinsb) [#6707](https://github.com/kubernetes/kops/pull/6707)
* Publish utils.tar.gz to github releases also [@justinsb](https://github.com/justinsb) [#6680](https://github.com/kubernetes/kops/pull/6680)
* Allow uneven etcd zones [@adammw](https://github.com/adammw) [#6641](https://github.com/kubernetes/kops/pull/6641)
* Add terraform support for additional CIDR blocks. [@rdrgmnzs](https://github.com/rdrgmnzs) [#6693](https://github.com/kubernetes/kops/pull/6693)
* Canal manifest updates for k8s v1.12+ [@KashifSaadat](https://github.com/KashifSaadat) [#6823](https://github.com/kubernetes/kops/pull/6823)
* Update to etcd-manager 1.0.20190509 [@justinsb](https://github.com/justinsb) [#6917](https://github.com/kubernetes/kops/pull/6917)
* S3 VFS: Default to current region from metadata service [@justinsb](https://github.com/justinsb),[@granular-ryanbonham](https://github.com/granular-ryanbonham) [#6943](https://github.com/kubernetes/kops/pull/6943)
* etcd-manager: Update to 3.0.20190513 [@justinsb](https://github.com/justinsb) [#6959](https://github.com/kubernetes/kops/pull/6959)
* Fix Docker not being installed on Ubuntu 16.04 [@meeee](https://github.com/meeee) [#6965](https://github.com/kubernetes/kops/pull/6965)
* Issue #6945 [@pkutishch](https://github.com/pkutishch),[@mikesplain](https://github.com/mikesplain) [#6951](https://github.com/kubernetes/kops/pull/6951)

# 1.12.0 to 1.12.1

* Don't panic when deleting instancegroups [@justinsb](https://github.com/justinsb) [#7000](https://github.com/kubernetes/kops/pull/7000)
* etcd-manager: update to 3.0.20190516 [@justinsb](https://github.com/justinsb) [#7007](https://github.com/kubernetes/kops/pull/7007)
* Terraform: fix options field, should be spot_options [@kimxogus](https://github.com/kimxogus) [#6988](https://github.com/kubernetes/kops/pull/6988)

# 1.12.1 to 1.12.2

* Mark ENI 0 as delete_on_termination for LaunchTemplates [@granular-ryanbonham](https://github.com/granular-ryanbonham) [#7094](https://github.com/kubernetes/kops/pull/7094)

# 1.12.2 to 1.12.3

* Cherry pick of #7211: Use NodeAuthorizer config options instead of soely [@jacksontj](https://github.com/jacksontj) [#7232](https://github.com/kubernetes/kops/pull/7232)
* Cherry pick of #7219: Make an actual deep-copy of the state [@jacksontj](https://github.com/jacksontj) [#7235](https://github.com/kubernetes/kops/pull/7235)
* Upgrade Calico to 3.7.2 [@asincu](https://github.com/asincu) [#7051](https://github.com/kubernetes/kops/pull/7051)
* Update canal to 3.6.4, for TTA-2019-002 [@justinsb](https://github.com/justinsb) [#7275](https://github.com/kubernetes/kops/pull/7275)
* Bumping calico to 3.7.4. [@michalschott](https://github.com/michalschott) [#7249](https://github.com/kubernetes/kops/pull/7249)
* Cherry pick of #7185: Replace behavior for aws hostnameOverride [@jacksontj](https://github.com/jacksontj) [#7308](https://github.com/kubernetes/kops/pull/7308)
* Calico -> 3.7.4 for older versions [@justinsb](https://github.com/justinsb) [#7282](https://github.com/kubernetes/kops/pull/7282)
* Bump etcd-manager to 3.0.20190801 [@justinsb](https://github.com/justinsb) [#7349](https://github.com/kubernetes/kops/pull/7349)
* Warn/prevent if the version of etcd is unsupported with etcd-manager [@justinsb](https://github.com/justinsb) [#7340](https://github.com/kubernetes/kops/pull/7340)