When the PublicJWKS feature-flag is set, we expose the apiserver JWKS
document publicly (including enabling anonymous access). This is a
stepping stone to a more hardened configuration where we copy the JWKS
document to S3/GCS/etc.
Co-authored-by: John Gardiner Myers <jgmyers@proofpoint.com>
kube-apiserver doesn't expose the healthcheck via a dedicated
endpoint, instead relying on anonyomous-access being enabled. That
has previously forced us to enable the unauthenticated endpoint on
127.0.0.1:8080.
Instead we now run a small sidecar container, which
proxies /healthz and /readyz requests (only) adding appropriate
authentication using a client certificate.
This will also enable better load balancer checks in future, as these
have previously been hampered by the custom CA certificate.
Co-authored-by: John Gardiner Myers <jgmyers@proofpoint.com>
Previously when setting the external cloud controller manager
configuration the core components `kubelet`, `apiserver` and
`kubecontroller-manager` were configured to use the external cloud
controller manager. Without setting the feature flag
EnableExternalCloudController this lead to a cluster in which the
masters had the cloud controller taint
`node.cloudprovider.kubernetes.io/uninitialized` which prevents
essential pods, like dns-controller to not be scheduled and leaves a
cluster where worker nodes can't connect to the api server because they
cannot resolve its hostname.
Image names from 1.16 on include an architecture suffix,
e.g. "-amd64"; the generic alias continues to work when pulling, but
when loading from a tarball (i.e. running in CI) we must use the
per-architecture name.
a) The current implementation use's a static kubelet which doesn't not conform to the Node authorization mode (i.e. system:nodes:<nodename>)
b) As present the kubeconfig is static and reused across all the masters and nodes
The PR firstly introduces the ability for users to use bootstrap tokens and secondly when enabled ensure the kubelets for the masters as have unique usernames. Note, this PR does not attempt to address the distribution of the bootstrap tokens themselves, that's for cluster admins. One solution for this would be a daemonset on the masters running on hostNetwork and reuse dns-controller to annotated the pods and give as the DNS
Notes:
- the master node do not use bootstrap tokens, instead given they have access to the ca anyhow, we generate certificates for each.
- when bootstrap token is not enabled the behaviour will stay the same; i.e. a kubelet configuration brought down from the store.
- when bootstrap tokens are enabled, the Nodes sit in a timeout loop waiting for the configuration to appear (by third party).
- given the nodeup docker and manifests builders are executed before the kubelet builder, the assumption here is a unit file kicks of a custom container to bootstrap the rest.
- the current firewalls on between the master and nodes are fairly open so no need to open ports between the two
- much of the work was ported from @justinsb PR [here](https://github.com/kubernetes/kops/pull/4134/)
- we add a very presumptuous server and client certificates for use with an authorizer (node-bootstrap-internal.dns_zone)
I do have an additional PR which performs the entire thing. The process being a node_authorizer which runs on the master nodes via a daemonset, the service implements a series of authorizers (i.e. alwaysallow, aws, gce etc). For aws, the process is similar to how vault authorizes nodes [here](https://www.vaultproject.io/docs/auth/aws.html). Nodeup no then calls out to the node_authorizer on bootstrap and provisions the kubelet.
Kubernetes 1.9 changed the default for etcd-quorum-read flag value to
true, in the hope of fixing some of the edge-case controller issues.
However, while this is cheap on etcd3, that fix was not backported to
etcd2, and performance there of quorum reads is poor.
For non-HA clusters with etcd2, it still goes through raft, but does not
need to - we set etcd-quorum-read to false, as this is just a missed
optimization in etcd2.
For HA clusters with etcd2, it's trickier, but at least for now we're
going to avoid the (crippling) performance regression. kops 1.10 should
have etcd-manager (allowing upgrades to etcd3), and the ability to
configure IOPS on the etcd volume, so we can revisit this in 1.10 /
1.11.
Automatic merge from submit-queue.
Initial aggregation work
Create the keypairs, which are supposed to be signed by a different CA.
Set the `--requestheader-...` flags on apiserver.
Fix#3152Fix#2691
- removing the StorageType on the etcd cluster spec (sticking with the Version field only)
- changed the protokube flag back to -etcd-image
- users have to explicitly set the etcd version now; the latest version in gcr.io is 3.0.17
- reverted the ordering on the populate spec
The current implementation is running v2.2.1 which is two year old and end of life. This PR add the ability to use etcd and set the versions if required. Note at the moment the image is still using the gcr.io registry image. As note, much like TLS their presently is not 'automated' migration path from v2 to v3.
- the feature is gated behine the storageType of the etcd cluster, bot clusters events and main must use the same storage type
- the version for v2 is unchanged and pinned at v2.2.1 with v2 using v3.0.17
- @question: we shoudl consider allowing the use to override the images though I think this should be addresses more generically, than one offs here and then. I know chris is working on a asset registry??
John Gardiner Myers
Ole Markus With
Justin SB
Ciprian Hacman
Kubernetes Prow Robot
Michael Wagner
mikesplain
Jesse Haka
Lily
Derek Lemon -T (delemon - AEROTEK INC at Cisco)
Jordan Liggitt
Rohith
Andrew Sy Kim
Justin Santa Barbara
Ripta Pasay
Davanum Srinivas
Blake
zengchen1024
Kubernetes Submit Queue
wlan0
georgebuckerfield
chrislovecnm