- add a new kops secret command to store encryption config
- add the experimential-encryption-provider-config flag to the kube-apiserver
- add functionality to nodeup to create the stored encryption config in the correct path
The current implementation does not place any limitation on the dns annontation which the dns-controller can consume. In a multi-tenented environment was have to ensure certain safe guards are met, so users can't byt accident or intentionally alter our internal dns. Note; the current behaviour has not been changed;
- added the --watch-namespace option to the dns controller and WatchNamespace to the spec
- cleaned up area of the code where possible or related
- fixed an vetting issues that i came across on the journey
- renamed the dns-controller watcher files
This is part I of multi part Pull Request to enable the
cloud-controller-manager through kops. This specific PR introduces the
cloud-controller-manager api, and puts it behind a feature flag.
Please feel free to merge this.
Automatic merge from submit-queue. .
Support additional config options for Canal Networking
Add support for additional global and iptables configuration options within the Canal Networking Spec: https://docs.projectcalico.org/v2.4/reference/felix/configuration
- **ChainInsertMode:** Controls whether Felix inserts rules to the top of iptables chains, or appends to the bottom. Leaving the default option is safest to prevent accidentally breaking connectivity. Default: 'insert' (other options: 'append')
- **PrometheusMetricsEnabled:** Set to enable the experimental Prometheus metrics server (default: false)
- **PrometheusMetricsPort:** TCP port that the experimental Prometheus metrics server should bind to (default: 9091)
- **PrometheusGoMetricsEnabled:** Enable Prometheus Go runtime metrics collection
- **PrometheusProcessMetricsEnabled:** Enable Prometheus process metrics collection
Automatic merge from submit-queue. .
Allow specifying a SSH key name for AWS
Related to #2309, this allows naming an existing key pair using the
cluster spec field `sshKeyName`.
In our use case, kops can now be used without providing the ability to
create EC2 key pairs.
Automatic merge from submit-queue
Add romana to built-in CNI options
This PR adds `romana` as a networking option for kops.
It installs the latest "preview" release of Romana v2.0, which provides the expected features in terms of IP allocations and route configuration. Network policy features are being ported to 2.0 and will be in the final release. (We intend to submit a followup PR for kops as part of that rolling out that release.)
Note: in this setup, we're using the etcd cluster that kops deploys for k8s. This isn't ideal, but some possibilities (eg: StatefulSets) aren't practical for the CNI itself, and creating a parallel etcd cluster via manifests seemed to be a more-intrusive approach than using the existing one.
If this is a concern or problem, then I'm very open to discussing and implementing it based on your suggestions.
Also, some functionality is exclusive to AWS environments. Other cloud platforms are on Romana's roadmap but not developed yet. Let me know that restriction needs to be enforced in code or directly documented.
Automatic merge from submit-queue
Added ability to set --service-node-port-range
Addresses: #3052 and #3326
From what I can tell, `--service-node-port-range` has been a valid option forever, so I haven't worried about k8s versions.
@justinsb @chrislovecnm let me know if any changes are required.
Automatic merge from submit-queue
Flannel: change default backend type
We support udp, which has to the default for backwards-compatibility,
but also new clusters will now use vxlan.
Related to #2309, this allows naming an existing key pair using the
cluster spec field `sshKeyName`.
In our use case, kops can now be used without providing the ability to
create EC2 key pairs.
Automatic merge from submit-queue
Create Keyset API type
A Keyset holds a set of keypairs or other secret cluster material.
It is a set to support rotation of keys.
This will allow us to store secrets on kops-server (and also is a step towards
separating where we manage secrets from how we communicate them to running
clusters, which will allow bare-metal or KMS)
Starting with just the API objects.
This will allow us to set CIDRs for nodeport access, which in turn will
allow e2e tests that require nodeport access to pass.
Then add a feature-flagged flag to `kops create cluster` to allow
arbitrary setting of spec values; currently the only value supported is
cluster.spec.nodePortAccess
Automatic merge from submit-queue
Adds DNSControllerSpec and WatchIngress flag
This PR is in reference to #2496, #2468 and the issues referenced in there relating to use of the watch-ingress flag.
This PR attempts to rectify this situation and gives users who want it, the option to turn on watch-ingress without forcing it on them. Also spits out a warning to the logs about potential side effects.
Includes notes in `docs/cluster_spec.md` to explain.
Automatic merge from submit-queue
Additional Kubelet Options
This PR add additional options to the kubelet spec allowing users to set the --runtime-request-timeout and -volume-stats-agg-period
In related to issue https://github.com/kubernetes/kops/issues/3265
Automatic merge from submit-queue
Kubelet Readonly Port
The current implementation does not permit the user to specify the kubelet read-only port (which unset defaults to 10255). For security reasons we need this port switched off i.e. 0. This PR retains the default behavior but adds the readOnlyPort as an option for those whom need to override.
```shell
podInfraContainerImage: gcr.io/google_containers/pause-amd64:3.0
podManifestPath: /etc/kubernetes/manifests
+ readOnlyPort: 0
registerSchedulable: false
requireKubeconfig: true
```
And tested on the box
```shell
core@ip-10-250-34-23 ~ $ egrep -o 'read-only-port=[0-9]+' /etc/sysconfig/kubelet
read-only-port=0
```
The current implementaton does not permit the user to specify the kubelet read-only port (which unset defaults to 10255). Note security reasons we need this port switched off i.e. 0. This PR retains the default behaviour but adds the readOnlyPort as an options for those whom need to override.
podInfraContainerImage: gcr.io/google_containers/pause-amd64:3.0
podManifestPath: /etc/kubernetes/manifests
+ readOnlyPort: 0
registerSchedulable: false
requireKubeconfig: true
Automatic merge from submit-queue
Allow user defined endpoint to host action for Canal
Adds ability to define `Networking.Canal.DefaultEndpointToHostAction` in the Cluster Spec. This allows you to customise the behaviour of traffic routing from a pod to the host (after calico iptables chains have been processed). `ACCEPT` is the default value and is left as-is.
`If you want to allow some or all traffic from endpoint to host, set this parameter to “RETURN” or “ACCEPT”. Use “RETURN” if you have your own rules in the iptables “INPUT” chain; Calico will insert its rules at the top of that chain, then “RETURN” packets to the “INPUT” chain once it has completed processing workload endpoint egress policy.`
Automatic merge from submit-queue
Docker Default Ulimits
The current implementation does not permit us to set the default ulimit on docker daemon (currently a requirement for our elasticsearch). This PR add the DefaultUlimit option to the DockerConfig
The current implementation does not permit us to set the default ulimit on docker daemon (currently a requirement for our logstash). This PR add the DefaultUlimit option to the DockerConfig
Automatic merge from submit-queue
Allow the strict IAM policies to be optional
The stricter IAM policies could potentially cause regression for some edge-cases, or may rely on nodeup image changes that haven't yet been deployed / tagged officially (currently the case on master branch since PR https://github.com/kubernetes/kops/pull/3158 was merged in).
This PR just wraps the new IAM policy rules around a cluster spec flag, `EnableStrictIAM`, so will default to the original behaviour (where the S3 policies were completely open). Could also be used to wrap PR https://github.com/kubernetes/kops/pull/3186 if it progresses any further.
- Or we could reject this and have the policies always strict! :)
Automatic merge from submit-queue
Cluster / InstanceGroup File Assets
@chrislovecnm @justinsb ...
The current implementation does not make it ease to fully customize nodes before kube install. This PR adds the ability to include file assets in the cluster and instaneGroup spec which can be consumed by nodeup. Allowing those whom need (i.e. me :-)) greater flexibilty around their nodes. @Note, nothing is enforced, so unless you've specified anything everything is as the same
- updated the cluster_spec.md to reflect the changes
- permit users to place inline files into the cluster and instance group specs
- added the ability to template the files, the Cluster and InstanceGroup specs are passed into context
- cleaned up and missed comment, unordered imports etc along the journey
notes: In addition to this; need to look at the detecting the changes in the cluster and instance group spec. Think out loud perhaps using a last_known_configuration annotation, similar to kubernetes
Automatic merge from submit-queue
Create cluster requirements for DigitalOcean
Initial changes required to create a cluster state. Running `kops update cluster --yes` does not work yet.
Note that DO has already adopted cloud controller managers (https://github.com/digitalocean/digitalocean-cloud-controller-manager) so we set `--cloud-provider=external`. This will end up being the case for aws, gce and vsphere over the next couple of releases.
https://github.com/kubernetes/kops/issues/2150
```bash
$ kops create cluster --cloud=digitalocean --name=dev.asykim.com --zones=tor1
I0821 18:47:06.302218 28623 create_cluster.go:845] Using SSH public key: /Users/AndrewSyKim/.ssh/id_rsa.pub
I0821 18:47:06.302293 28623 subnets.go:183] Assigned CIDR 172.20.32.0/19 to subnet tor1
Previewing changes that will be made:
I0821 18:47:11.457696 28623 executor.go:91] Tasks: 0 done / 27 total; 27 can run
I0821 18:47:12.113133 28623 executor.go:91] Tasks: 27 done / 27 total; 0 can run
Will create resources:
Keypair/kops
Subject o=system:masters,cn=kops
Type client
Keypair/kube-controller-manager
Subject cn=system:kube-controller-manager
Type client
Keypair/kube-proxy
Subject cn=system:kube-proxy
Type client
Keypair/kube-scheduler
Subject cn=system:kube-scheduler
Type client
Keypair/kubecfg
Subject o=system:masters,cn=kubecfg
Type client
Keypair/kubelet
Subject o=system:nodes,cn=kubelet
Type client
Keypair/kubelet-api
Subject cn=kubelet-api
Type client
Keypair/master
Subject cn=kubernetes-master
Type server
AlternateNames [100.64.0.1, 127.0.0.1, api.dev.asykim.com, api.internal.dev.asykim.com, kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local]
ManagedFile/dev.asykim.com-addons-bootstrap
Location addons/bootstrap-channel.yaml
ManagedFile/dev.asykim.com-addons-core.addons.k8s.io
Location addons/core.addons.k8s.io/v1.4.0.yaml
ManagedFile/dev.asykim.com-addons-dns-controller.addons.k8s.io-k8s-1.6
Location addons/dns-controller.addons.k8s.io/k8s-1.6.yaml
ManagedFile/dev.asykim.com-addons-dns-controller.addons.k8s.io-pre-k8s-1.6
Location addons/dns-controller.addons.k8s.io/pre-k8s-1.6.yaml
ManagedFile/dev.asykim.com-addons-kube-dns.addons.k8s.io-k8s-1.6
Location addons/kube-dns.addons.k8s.io/k8s-1.6.yaml
ManagedFile/dev.asykim.com-addons-kube-dns.addons.k8s.io-pre-k8s-1.6
Location addons/kube-dns.addons.k8s.io/pre-k8s-1.6.yaml
ManagedFile/dev.asykim.com-addons-limit-range.addons.k8s.io
Location addons/limit-range.addons.k8s.io/v1.5.0.yaml
ManagedFile/dev.asykim.com-addons-storage-aws.addons.k8s.io
Location addons/storage-aws.addons.k8s.io/v1.6.0.yaml
Secret/admin
Secret/kube
Secret/kube-proxy
Secret/kubelet
Secret/system:controller_manager
Secret/system:dns
Secret/system:logging
Secret/system:monitoring
Secret/system:scheduler
Must specify --yes to apply changes
Cluster configuration has been created.
Suggestions:
* list clusters with: kops get cluster
* edit this cluster with: kops edit cluster dev.asykim.com
* edit your node instance group: kops edit ig --name=dev.asykim.com nodes
* edit your master instance group: kops edit ig --name=dev.asykim.com master-tor1
Finally configure your cluster with: kops update cluster dev.asykim.com --yes
```
This enables external admission controller webhooks, api aggregation,
and anything else that relies on the
--proxy-client-cert-file/--proxy-client-key-file apiserver args.
- removed the Mode field from the FileAsset spec
- removed the ability to template the content
- removed the need to specify the Path and instead default to /srv/kubernetes/assets/<name>
- change the FileAssets from []*FileAssets to []FileAssets
The current implementation does not make it ease to fully customize nodes before kube install. This PR adds the ability to include file assets in the cluster and instaneGroup spec which can be consumed by nodeup. Allowing those whom need (i.e. me :-)) greater flexibilty around their nodes. @Note, nothing is enforced, so unless you've specified anything everything is as the same
- updated the cluster_spec.md to reflect the changes
- permit users to place inline files into the cluster and instance group specs
- added the ability to template the files, the Cluster and InstanceGroup specs are passed into context
- cleaned up and missed comment, unordered imports etc along the journey
Automatic merge from submit-queue
Cluster Hooks Enhancement
Cluster Hook Enhancement
The current implementation is presently limited to docker exec, without ordering or any bells and whistles. This PR extends the functionality of the hook spec by;
- adds ordering to the hooks, with users able to set the requires and before of the unit
- cleaned up the manifest code, added tests and permit setting a section raw
- added the ability to filter hooks via master and node roles
- updated the documentation to reflect the changes
- extending the hooks to permit adding hooks per instancegroup as well cluster
- @note, instanceGroup are permitted to override the cluster wide one for ease of testing
- on the journey tried to fix an go idioms such as import ordering, comments for global export etc
- @question: v1alpha1 doesn't appear to have Subnet fields, are these different version being used anywhere?
- removing the StorageType on the etcd cluster spec (sticking with the Version field only)
- changed the protokube flag back to -etcd-image
- users have to explicitly set the etcd version now; the latest version in gcr.io is 3.0.17
- reverted the ordering on the populate spec
The current implementation is running v2.2.1 which is two year old and end of life. This PR add the ability to use etcd and set the versions if required. Note at the moment the image is still using the gcr.io registry image. As note, much like TLS their presently is not 'automated' migration path from v2 to v3.
- the feature is gated behine the storageType of the etcd cluster, bot clusters events and main must use the same storage type
- the version for v2 is unchanged and pinned at v2.2.1 with v2 using v3.0.17
- @question: we shoudl consider allowing the use to override the images though I think this should be addresses more generically, than one offs here and then. I know chris is working on a asset registry??
- switched to using an array of roles rather than boolean flags for node selection
- fixed up the README to reflect the changes
- added the docker.service as a Requires to all docker exec hooks
- extending the hooks to permit adding hooks per instancegroup as well
- @note, instanceGroup are permitted to override the cluster wide one for ease of testing
- updated the documentation to reflect the changes
- on the journey tried to fix an go idioms such as import ordering, comments for global export etc
- @question: v1alpha1 doesn't appear to have Subnet fields, are these different version being used anywhere?
The present implementation of hooks only perform for docker exec, which isn't that flexible. This PR permits the user to greater customize systemd units on the instances
- cleaned up the manifest code, added tests and permit setting a section raw
- added the ability to filter hooks via master and node roles
- updated the documentation to reflect the changes
- cleaned up some of the vetting issues
The current implementation does not permit the user to order the hooks. This PR adds optional Requires, Before and Documentation to the HookSpec which is added the systemd unit if specified.
Another step towards working totally offline (which may never be fully
achievable, because of the need to hash assets). But should ensure that
when we update the stable channel, we are testing against that version
in the tests, otherwise it is easy to break master.
- added the master option back the protokube, updating the nodeup model and protokube code
- removed any comments no related to the PR as suggested
- reverted the ordering of the mutex in the AWSVolumes in protokube
The current implementation does not put any transport security on the etcd cluster. The PR provides and optional flag to enable TLS the etcd cluster
- cleaned up and fixed any formatting issues on the journey
- added two new certificates (server/client) for etcd peers and a client certificate for kubeapi and others perhaps (perhaps calico?)
- disabled the protokube service for nodes completely is not required; note this was first raised in https://github.com/kubernetes/kops/pull/3091, but figured it would be easier to place in here given the relation
- updated protokube codebase to reflect the changes, removing the master option as its no longer required
- added additional integretion tests for the protokube manifests;
- note, still need to add documentation, but opening the PR to get feedback
- one outstanding issue is the migration from http -> https for preexisting clusters, i'm gonna hit the coreos board to ask for the best options
Automatic merge from submit-queue
Vetting / Formatting / Cleanup
- fixed any of the vettting / formatting issues that i'm came across on the update
- removed the commented out lines from the componentconfig as it make its increasingly difficult to find what is supported, what is not and the difference between them.
- added SerializeImagePulls, RegisterSchedulable to kubelet (by default they are ignored)
- added FeatureGates to the kube-proxy
- fixed the vetting issue with NodeUpConfig -> Config
- added the aws london regions for cloud provider lookup
Out of interest can someone point me to where these multi-versioned componentconfig are being used?
- fixed any of the vettting / formatting issues that i'm came across on the update
- removed the commented out lines from the componentconfig as it make its increasingly difficult to find what is supported, what is not and the difference between them.
- added SerializeImagePulls, RegisterSchedulable to kubelet (by default they are ignored)
- added FeatureGates to the kube-proxy
Out of interest can someone point me to where these multi-versioned componentconfig are being used?
- adding the options to permit adjusting the cluster to use auth on the kubelet
- for testing i've used kubelet-client-certificate and kubelet-client-key to /srv/kubernetes/server.{cert,key} and
setting the --client-ca-file on the node kubelet to /srv/kubernetes/ca.crt (tested as working).
- note i'm not enforcing anything i.e. the user has to edit the cluster and apply the configuration on the kubelet and kubeAPIServer in the cluster spec
We modelled our VFS clientset (for API objects backed by a VFS path)
after the "real" clientsets, so now it is relatively easy to add a
second implementation that will be backed by a real clientset.
The snafu here is that we weren't really using namespaces previously.
Namespaces do seem to be the primary RBAC scoping mechanism though, so
we start using them with the real clientset.
The namespace is currently inferred from the cluster name. We map dots
to dashes, because of namespace limitations, which could yield
collisions, but we'll deal with this by simply preventing users from
creating conflicting cluster names - i.e. you simply won't be able to
create a.b.example.com and a-b.example.com
fixes#2606
Most part of the changes are similar to current supported CNI networking
provider. Kube-router also support IPVS bassed service proxy which can
be used as replacement for kube-proxy. So the manifest for kube-router
included with this patch enables kube-router to provide pod-to-pod
networking, IPVS based service proxy and ingress pod firewall.
This gives the ability to a user to enable cross-subnet mode in Calico.
Also introduces a new addon that, full disclosure, I wrote.
[ottoyiu/k8s-ec2-srcdst](https://github.com/ottoyiu/k8s-ec2-srcdst)
* Add support of CoreDNS for vSphere provider.
* Add instructions about how to setup CoreDNS for vSphere provider.
* Address comments for CoreDNS support code.
- new property is only used when KubernetesVersion is 1.6 or greater
- taints are passed to kubelet via --register-with-taints flag
- Set a default NoSchedule taint on masters
- Set --register-schedule=true when --register-with-taints is used
- Changed the log message in taints.go to be less alarming if taints are
found - since they are expected on 1.6.0+ clusters
- Added Taints section to the InstanceGroup docs
- Only default taints are allowed in the spec pre-1.6
- Custom taint validation happens as soon as IG specs are edited.
- Previous method would have caused issues with the way tags are used
for filtering resources.
- Updated docs and comments to only refer to instance groups, rather
than all AWS resources
- --cloud-labels will be applied to every kops-created resource
- Also ran apimachinery to regenerated the conversions for the new
Cluster.ClusterLabels property.
* Detect CoreOS
* Move key manifests to code, to tolerate read-only mounts
* Misc refactorings so more code can be shared
* Change lots of ints to int32s in the models
* Run nodeup as a oneshot systemd service, rather than relying on
cloud-init behaviour which varies across distros
* Integrating Canal (Flannel + Calico) for CNI
Initial steps to integrate Canal as a CNI provider for kops
Removed CNI in help as per chrislovecnm
* Integration tests, getting closer to working
- Added some integration tests for Canal
- Finding more places Canal needed to be added
- Sneaking in update to Calico Policy Controller
* Add updated conversion file
* turned back on canal integration tests
* fixed some rebase issues
* Fixed tests and flannel version
* Fixed canal yaml, and some rebasing errors
- Added some env vars to the install-cni container to get the proper
node name handed off
* Added resource limits
- set resource limits on containers for Canal
- Ran through basic calico tutorials to verify functionality
* Updating Calico parts to Calico 2.0.2
georgebuckerfield
Rohith
wlan0
Kubernetes Submit Queue
Kashif Saadat
Andrey Chernih
Caleb Gilmour
Justin Santa Barbara
Robin Percy
John Zeringue
Eric Hole
Alexander Brandstedt
Torin Sandall
andrewsykim
asifdxtreme
Derek VerLee
Ionut Craciunescu
Phillip Wittrock
Chris Love
King'ori Maina
Hanfei Shen
Austin Moore
alexandrst88
Murali Reddy
Jordan Jennings
Otto Yiu
Abrar Shivani
Miao Luo
Kris Nova
Robin Percy
Leon Waldman
Daniel Cohen
Faraaz Khan
Seth Pollack
Michael Taufen
yissachar
jg Chen
Seth Pollack
Matt Sanford
fate-grand-order
Zac Blazic
Matthew Mihok
Manuel de Brito Fontes
Stephen Schlie