a) The current implementation use's a static kubelet which doesn't not conform to the Node authorization mode (i.e. system:nodes:<nodename>)
b) As present the kubeconfig is static and reused across all the masters and nodes
The PR firstly introduces the ability for users to use bootstrap tokens and secondly when enabled ensure the kubelets for the masters as have unique usernames. Note, this PR does not attempt to address the distribution of the bootstrap tokens themselves, that's for cluster admins. One solution for this would be a daemonset on the masters running on hostNetwork and reuse dns-controller to annotated the pods and give as the DNS
Notes:
- the master node do not use bootstrap tokens, instead given they have access to the ca anyhow, we generate certificates for each.
- when bootstrap token is not enabled the behaviour will stay the same; i.e. a kubelet configuration brought down from the store.
- when bootstrap tokens are enabled, the Nodes sit in a timeout loop waiting for the configuration to appear (by third party).
- given the nodeup docker and manifests builders are executed before the kubelet builder, the assumption here is a unit file kicks of a custom container to bootstrap the rest.
- the current firewalls on between the master and nodes are fairly open so no need to open ports between the two
- much of the work was ported from @justinsb PR [here](https://github.com/kubernetes/kops/pull/4134/)
- we add a very presumptuous server and client certificates for use with an authorizer (node-bootstrap-internal.dns_zone)
I do have an additional PR which performs the entire thing. The process being a node_authorizer which runs on the master nodes via a daemonset, the service implements a series of authorizers (i.e. alwaysallow, aws, gce etc). For aws, the process is similar to how vault authorizes nodes [here](https://www.vaultproject.io/docs/auth/aws.html). Nodeup no then calls out to the node_authorizer on bootstrap and provisions the kubelet.
The current implementation doesn't work with PodSecurityPolicies enabled due to no psp policies. This PR adds a default psp policy for the kubelet users and and the kube-system namespace
Kubernetes 1.9 changed the default for etcd-quorum-read flag value to
true, in the hope of fixing some of the edge-case controller issues.
However, while this is cheap on etcd3, that fix was not backported to
etcd2, and performance there of quorum reads is poor.
For non-HA clusters with etcd2, it still goes through raft, but does not
need to - we set etcd-quorum-read to false, as this is just a missed
optimization in etcd2.
For HA clusters with etcd2, it's trickier, but at least for now we're
going to avoid the (crippling) performance regression. kops 1.10 should
have etcd-manager (allowing upgrades to etcd3), and the ability to
configure IOPS on the etcd volume, so we can revisit this in 1.10 /
1.11.
Automatic merge from submit-queue.
Add audit log format flag for api server
In kubernetes 1.8, a flag for audit-log format has been added, this flag can be set to `legacy` or `json` format on the kubernetes API server.
Automatic merge from submit-queue.
Support for OIDC 'username-prefix' and 'groups-prefix' flags
### What
Added support for `--oidc-username-prefix` and `--oidc-groups-prefix`.
By passing these it's possible to override the default prefixes used to map the OIDC user with the username in kubernetes.
### See
See: https://kubernetes.io/docs/admin/authentication/#configuring-the-api-server
### IMPORTANT
I'm far from a kubernetes/KOPS, this is not tested so someone needs to have a look and see if something is missing or if this can cause troubles! (don't want to accidentally cause the destruction of the universe 💥 )
It's basically the same done in this other PR: https://github.com/kubernetes/kops/pull/1438/files
I did **not** change the `zz_generated.conversion.go` files as according to comment at the top of them they're autogenerated:
```go
// This file was autogenerated by conversion-gen. Do not edit it manually!
```
(I wonder if they should be `.gitignore`d)
### Fixes
This should fix [#4007: field oidcUsernamePrefix is not recognized in cluster configuration file](https://github.com/kubernetes/kops/issues/4007)
Automatic merge from submit-queue.
Allow disabling kube-proxy
This adds the ability to turn off kube-proxy.
My specific use case is the usage of a custom CNI.
Automatic merge from submit-queue.
Initial aggregation work
Create the keypairs, which are supposed to be signed by a different CA.
Set the `--requestheader-...` flags on apiserver.
Fix#3152Fix#2691
Automatic merge from submit-queue. .
Support encryption-at-rest for the kube-apiserver
This PR adds support for enabling encryption-at-rest for data in etcd, via the kube-apiserver (as per https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data).
I've put the functionality behind a feature flag, `+EnableDataEncryption`. It can then be enabled per-cluster by using `--enable-encryption-config` on the command line, or by adding a `kubeEncryptionConfig` section to the cluster spec. This is passed through to the kube-apiserver by the nodeup process. I'm not sure if this is the best way of doing it right now, but it is working.
Fixes#3356.
- add a new kops secret command to store encryption config
- add the experimential-encryption-provider-config flag to the kube-apiserver
- add functionality to nodeup to create the stored encryption config in the correct path
This is part I of multi part Pull Request to enable the
cloud-controller-manager through kops. This specific PR introduces the
cloud-controller-manager api, and puts it behind a feature flag.
Please feel free to merge this.
Automatic merge from submit-queue
Additional Kubelet Options
This PR add additional options to the kubelet spec allowing users to set the --runtime-request-timeout and -volume-stats-agg-period
In related to issue https://github.com/kubernetes/kops/issues/3265
The current implementaton does not permit the user to specify the kubelet read-only port (which unset defaults to 10255). Note security reasons we need this port switched off i.e. 0. This PR retains the default behaviour but adds the readOnlyPort as an options for those whom need to override.
podInfraContainerImage: gcr.io/google_containers/pause-amd64:3.0
podManifestPath: /etc/kubernetes/manifests
+ readOnlyPort: 0
registerSchedulable: false
requireKubeconfig: true
This enables external admission controller webhooks, api aggregation,
and anything else that relies on the
--proxy-client-cert-file/--proxy-client-key-file apiserver args.
- added the master option back the protokube, updating the nodeup model and protokube code
- removed any comments no related to the PR as suggested
- reverted the ordering of the mutex in the AWSVolumes in protokube
The current implementation does not put any transport security on the etcd cluster. The PR provides and optional flag to enable TLS the etcd cluster
- cleaned up and fixed any formatting issues on the journey
- added two new certificates (server/client) for etcd peers and a client certificate for kubeapi and others perhaps (perhaps calico?)
- disabled the protokube service for nodes completely is not required; note this was first raised in https://github.com/kubernetes/kops/pull/3091, but figured it would be easier to place in here given the relation
- updated protokube codebase to reflect the changes, removing the master option as its no longer required
- added additional integretion tests for the protokube manifests;
- note, still need to add documentation, but opening the PR to get feedback
- one outstanding issue is the migration from http -> https for preexisting clusters, i'm gonna hit the coreos board to ask for the best options
- fixed any of the vettting / formatting issues that i'm came across on the update
- removed the commented out lines from the componentconfig as it make its increasingly difficult to find what is supported, what is not and the difference between them.
- added SerializeImagePulls, RegisterSchedulable to kubelet (by default they are ignored)
- added FeatureGates to the kube-proxy
Out of interest can someone point me to where these multi-versioned componentconfig are being used?
- adding the options to permit adjusting the cluster to use auth on the kubelet
- for testing i've used kubelet-client-certificate and kubelet-client-key to /srv/kubernetes/server.{cert,key} and
setting the --client-ca-file on the node kubelet to /srv/kubernetes/ca.crt (tested as working).
- note i'm not enforcing anything i.e. the user has to edit the cluster and apply the configuration on the kubelet and kubeAPIServer in the cluster spec
* Add support of CoreDNS for vSphere provider.
* Add instructions about how to setup CoreDNS for vSphere provider.
* Address comments for CoreDNS support code.
- new property is only used when KubernetesVersion is 1.6 or greater
- taints are passed to kubelet via --register-with-taints flag
- Set a default NoSchedule taint on masters
- Set --register-schedule=true when --register-with-taints is used
- Changed the log message in taints.go to be less alarming if taints are
found - since they are expected on 1.6.0+ clusters
- Added Taints section to the InstanceGroup docs
- Only default taints are allowed in the spec pre-1.6
- Custom taint validation happens as soon as IG specs are edited.
Otherwise we were not evicting based on low inodes
Also add the notion of a flag-default, so we can pass fewer spurious
flags, and gget closer to the component model
* Zones are now subnets
* Utility subnet is no longer part of Zone
* Bastion InstanceGroup type added instead
* Etcd clusters defined in terms of InstanceGroups, not zones
* AdminAccess split into SSHAccess & APIAccess
* Dropped unused Multizone flag
Marc-Andre Belanger