Automatic merge from submit-queue.
Map docker 1.13.1 & 17.03.1, default 1.8 to 1.13.1
Kubernetes 1.8 adds validation for 1.13.1 and 17.03.1 (17.03.1 being
essentially the same as 1.13.1).
For 1.8, the default should be 1.13.1
Automatic merge from submit-queue.
Initial bazel support
Builds on the 1.8 version bump
The "trick" is to strip the BUILD & BUILD.bazel files from the vendor-ed deps.
Will rebase after 1.8 version bump merges.
Automatic merge from submit-queue.
Tweak signature in #3478
Follow-on to #3478
Because the default doesn't depend on the user-specified value, it's
misleading to pass it in.
Automatic merge from submit-queue.
Add Zones field to InstanceGroup
The Zones field can specify zones where they are not specified on a
Subnet, for example on GCE where we have regional subnets.
Automatic merge from submit-queue.
using same disk sizes for gce
Using the same disk size for gce and aws. We need to override the disk sizes in the e2e tests, but that is another PR.
Closes: https://github.com/kubernetes/kops/issues/3294
Automatic merge from submit-queue.
Refactoring to use cloud method for getting cloud groups
This builds on various other PRs. The only two SHAs to review are:
1. Moving delete instancesgroups into its own file d52d767508
2. Refactoring to use cloud based GetCloudGroups c33a078f52
AWS is the only one that has GetCloudGroups implemented at this point. GCE is next.
TODO
- [x] e2e testing rolling-update
- [x] e2e testing rolling-update with only one instance group
- [x] e2e testing force
- [x] e2e testing cloud-only
Updates
I have moved more of the code into `/pkg/cloudinstances` per guidance from @justinsb! I am liking it more!
The current kube manifest redirect all the logs into host located log files, this PR uses the tee command to pipe into both local logs (retaining the current) and docker stdout (which will be picked up by the journald or which every logging your using. Note also permits as to now need the logs via the kubectl command.
- renamed some of the files to make things cleaner
- redirecting the logs from the kubernetes components into local file and stdout
- cleaned up any vetting or linting error i came across
Automatic merge from submit-queue. .
promoting drain and validate by setting feature flag to true
I am unable to recreate https://github.com/kubernetes/kops/issues/2407, and frankly, it may be an edge case. We could warn a user if their wait times are low, but that would be another PR.
This PR moves Drain and Validate functionality for rolling-updates into the default user experience, setting the Feature Flag to true.
Per feedback, I am using the node and master interval times for the validation.
Automatic merge from submit-queue. .
Create GCE networks in auto mode, not legacy mode
auto mode allows for conversion to custom mode at the API level, and
legacy mode is deprecated.
Automatic merge from submit-queue. .
Support encryption-at-rest for the kube-apiserver
This PR adds support for enabling encryption-at-rest for data in etcd, via the kube-apiserver (as per https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data).
I've put the functionality behind a feature flag, `+EnableDataEncryption`. It can then be enabled per-cluster by using `--enable-encryption-config` on the command line, or by adding a `kubeEncryptionConfig` section to the cluster spec. This is passed through to the kube-apiserver by the nodeup process. I'm not sure if this is the best way of doing it right now, but it is working.
Fixes#3356.
Automatic merge from submit-queue. .
Add apimachinery changes to enable cloud controller manager
This is part I of multi part Pull Request to enable the
cloud-controller-manager through kops. This specific PR introduces the
cloud-controller-manager api, and puts it behind a feature flag.
Please feel free to merge this.
- add a new kops secret command to store encryption config
- add the experimential-encryption-provider-config flag to the kube-apiserver
- add functionality to nodeup to create the stored encryption config in the correct path
The current implementation does not place any limitation on the dns annontation which the dns-controller can consume. In a multi-tenented environment was have to ensure certain safe guards are met, so users can't byt accident or intentionally alter our internal dns. Note; the current behaviour has not been changed;
- added the --watch-namespace option to the dns controller and WatchNamespace to the spec
- cleaned up area of the code where possible or related
- fixed an vetting issues that i came across on the journey
- renamed the dns-controller watcher files
This is part I of multi part Pull Request to enable the
cloud-controller-manager through kops. This specific PR introduces the
cloud-controller-manager api, and puts it behind a feature flag.
Please feel free to merge this.
Automatic merge from submit-queue. .
Support additional config options for Canal Networking
Add support for additional global and iptables configuration options within the Canal Networking Spec: https://docs.projectcalico.org/v2.4/reference/felix/configuration
- **ChainInsertMode:** Controls whether Felix inserts rules to the top of iptables chains, or appends to the bottom. Leaving the default option is safest to prevent accidentally breaking connectivity. Default: 'insert' (other options: 'append')
- **PrometheusMetricsEnabled:** Set to enable the experimental Prometheus metrics server (default: false)
- **PrometheusMetricsPort:** TCP port that the experimental Prometheus metrics server should bind to (default: 9091)
- **PrometheusGoMetricsEnabled:** Enable Prometheus Go runtime metrics collection
- **PrometheusProcessMetricsEnabled:** Enable Prometheus process metrics collection
Automatic merge from submit-queue. .
Allow specifying a SSH key name for AWS
Related to #2309, this allows naming an existing key pair using the
cluster spec field `sshKeyName`.
In our use case, kops can now be used without providing the ability to
create EC2 key pairs.
Automatic merge from submit-queue
Add file assets to node user data scripts, fingerprint fileAssets and hooks content.
**Changes made:**
- Include FileAssets in the bootstrapscript (user-data for all nodes), selectively dependent on the roles specified for each asset.
- Fingerprint the sections of the FileAssets (Content) and Hooks (Manifests, ExecContainer Commands) Specs within the bootstrap script to reduce size (otherwise this can very quickly hit the 16KB user data limit with AWS).
Automatic merge from submit-queue
Revision to IAM Policies created by Kops
Based off of the work done by @chrislovecnm in PR #2497.
This PR tightens down the IAM policies created for Master & Node instance groups. The Cluster Spec `IAMSpec.Legacy` flag is used to control application of stricter policy rules, which is defaulted to true for existing clusters (to limit potential regression impact), and false for new cluster creation.
Automatic merge from submit-queue
Add romana to built-in CNI options
This PR adds `romana` as a networking option for kops.
It installs the latest "preview" release of Romana v2.0, which provides the expected features in terms of IP allocations and route configuration. Network policy features are being ported to 2.0 and will be in the final release. (We intend to submit a followup PR for kops as part of that rolling out that release.)
Note: in this setup, we're using the etcd cluster that kops deploys for k8s. This isn't ideal, but some possibilities (eg: StatefulSets) aren't practical for the CNI itself, and creating a parallel etcd cluster via manifests seemed to be a more-intrusive approach than using the existing one.
If this is a concern or problem, then I'm very open to discussing and implementing it based on your suggestions.
Also, some functionality is exclusive to AWS environments. Other cloud platforms are on Romana's roadmap but not developed yet. Let me know that restriction needs to be enforced in code or directly documented.
Automatic merge from submit-queue
add autoscaling:DescribeLaunchConfigurations permission
As of 0.6.1, Cluster Autoscaler supports [scaling node groups from/to 0](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler/cloudprovider/aws#scaling-a-node-group-to-0), but requires the `autoscaling:DescribeLaunchConfigurations` permission.
It'd be great to have this in kops since this permission needs to be re-added back to the master policy every time the cluster is updated.
Automatic merge from submit-queue
Added ability to set --service-node-port-range
Addresses: #3052 and #3326
From what I can tell, `--service-node-port-range` has been a valid option forever, so I haven't worried about k8s versions.
@justinsb @chrislovecnm let me know if any changes are required.
Automatic merge from submit-queue
Flannel: change default backend type
We support udp, which has to the default for backwards-compatibility,
but also new clusters will now use vxlan.
Automatic merge from submit-queue
Toolbox template
Extending the current implementation of toolbox template to include multiple files and snippets. Note, I've removed the requirements for defaults as I think people should be forced to specifically pass them
- allowing the users to use a snippets directory for reusable templates
- allows the users to specify multiple templates files via multiple --template <path>, use a directory or both
- allows the users to specify multiple configuration files via multiple --values <path>, use a directory or both
- adding a safety check to ensure templates don't reference an unknown values
- fixing the vetting issues to the method YamlToJson -> YAMLToJSON
- as usual anything a saw on the journey which doesn't comply with go-vet got changed
Examples of a snippet
```YAML
hooks:
- name: some_service.service
manifest: |
{{ include "some_service.service" . | indent 6 }}
```
We currently use something similar to template our cluster and instances group documents, handling the differences between prod, ci and ephemeral
Extending the current implementation of toolbox template to include multiple files and snippets. Note, i've removed the requirements for defaults as I think people should be forced to specifically pass them.
- fixing the vetting iseues to the method YamlToJson -> YAMLToJSON
- adding a safety check to ensure templates don't reference an unknown value
- extending the unit test to ensure the above works on main and snippets
- include the ability to specify multiple configuration files, useful for common.yaml and prod.yaml etc
Requested Changes - Toolbox Templating
Added the requested changes
- moved the templater into it's own package rather than using base util
- moved to using the sprig library for additional template function
- @note: i couldn't find a native way in sprig to do snippets, also the i've overloaded the indent as it appears to do the indent on all lines rather than on the newline, meaning i'd have to shift my first line back by the indent to get it to work, which seems ugly
Related to #2309, this allows naming an existing key pair using the
cluster spec field `sshKeyName`.
In our use case, kops can now be used without providing the ability to
create EC2 key pairs.
Automatic merge from submit-queue
Create Keyset API type
A Keyset holds a set of keypairs or other secret cluster material.
It is a set to support rotation of keys.
This will allow us to store secrets on kops-server (and also is a step towards
separating where we manage secrets from how we communicate them to running
clusters, which will allow bare-metal or KMS)
Starting with just the API objects.
This will allow us to set CIDRs for nodeport access, which in turn will
allow e2e tests that require nodeport access to pass.
Then add a feature-flagged flag to `kops create cluster` to allow
arbitrary setting of spec values; currently the only value supported is
cluster.spec.nodePortAccess
Automatic merge from submit-queue
Adds DNSControllerSpec and WatchIngress flag
This PR is in reference to #2496, #2468 and the issues referenced in there relating to use of the watch-ingress flag.
This PR attempts to rectify this situation and gives users who want it, the option to turn on watch-ingress without forcing it on them. Also spits out a warning to the logs about potential side effects.
Includes notes in `docs/cluster_spec.md` to explain.
Automatic merge from submit-queue
Additional Kubelet Options
This PR add additional options to the kubelet spec allowing users to set the --runtime-request-timeout and -volume-stats-agg-period
In related to issue https://github.com/kubernetes/kops/issues/3265
Automatic merge from submit-queue
Kubelet Readonly Port
The current implementation does not permit the user to specify the kubelet read-only port (which unset defaults to 10255). For security reasons we need this port switched off i.e. 0. This PR retains the default behavior but adds the readOnlyPort as an option for those whom need to override.
```shell
podInfraContainerImage: gcr.io/google_containers/pause-amd64:3.0
podManifestPath: /etc/kubernetes/manifests
+ readOnlyPort: 0
registerSchedulable: false
requireKubeconfig: true
```
And tested on the box
```shell
core@ip-10-250-34-23 ~ $ egrep -o 'read-only-port=[0-9]+' /etc/sysconfig/kubelet
read-only-port=0
```
The current implementaton does not permit the user to specify the kubelet read-only port (which unset defaults to 10255). Note security reasons we need this port switched off i.e. 0. This PR retains the default behaviour but adds the readOnlyPort as an options for those whom need to override.
podInfraContainerImage: gcr.io/google_containers/pause-amd64:3.0
podManifestPath: /etc/kubernetes/manifests
+ readOnlyPort: 0
registerSchedulable: false
requireKubeconfig: true
Automatic merge from submit-queue
Implementing GCE as an interface - modelling aws cloud provider
GCE and other cloud providers are structs instead of an interface. AWS cloud provider implements an interface. This PR refactors `GCECloud` as an interface, and creates `gceCloudImplementation`.
- [x] Need to e2e test
Automatic merge from submit-queue
Allow user defined endpoint to host action for Canal
Adds ability to define `Networking.Canal.DefaultEndpointToHostAction` in the Cluster Spec. This allows you to customise the behaviour of traffic routing from a pod to the host (after calico iptables chains have been processed). `ACCEPT` is the default value and is left as-is.
`If you want to allow some or all traffic from endpoint to host, set this parameter to “RETURN” or “ACCEPT”. Use “RETURN” if you have your own rules in the iptables “INPUT” chain; Calico will insert its rules at the top of that chain, then “RETURN” packets to the “INPUT” chain once it has completed processing workload endpoint egress policy.`
Automatic merge from submit-queue
Limit the IAM EC2 policy for the master nodes
Related to: https://github.com/kubernetes/kops/pull/3158
The EC2 policy for the master nodes are quite open currently, allowing them to create/delete/modify resources that are not associated with the cluster the node originates from. I've come up with a potential solution using condition keys to validate that the `ec2:ResourceTag/KubernetesCluster` matches the cluster name.
Automatic merge from submit-queue
Inline Component Configuration Fix
The current implementation does not ignore any possible interpolation of bash in the content. This PR wrapped the various spec content in 'EOF' to ignore all. All tested on a working cluster.
- updated the tests to reflect the changes
- wrapped the component configuration in 'eof' to ensure interpolation is ignored
- dropped the t.Log debug line
The current implementation does not ignore any possible interpolation of bash in the content. This PR wrapped the various spec content in 'EOF' to ignore all.
- updated the tests to reflect the changes
- wrapped the component configuration in 'eof' to ensure interpolation is ignored
Automatic merge from submit-queue
Docker Default Ulimits
The current implementation does not permit us to set the default ulimit on docker daemon (currently a requirement for our elasticsearch). This PR add the DefaultUlimit option to the DockerConfig
The current implementation does not permit us to set the default ulimit on docker daemon (currently a requirement for our logstash). This PR add the DefaultUlimit option to the DockerConfig
Automatic merge from submit-queue
Allow the strict IAM policies to be optional
The stricter IAM policies could potentially cause regression for some edge-cases, or may rely on nodeup image changes that haven't yet been deployed / tagged officially (currently the case on master branch since PR https://github.com/kubernetes/kops/pull/3158 was merged in).
This PR just wraps the new IAM policy rules around a cluster spec flag, `EnableStrictIAM`, so will default to the original behaviour (where the S3 policies were completely open). Could also be used to wrap PR https://github.com/kubernetes/kops/pull/3186 if it progresses any further.
- Or we could reject this and have the policies always strict! :)
Automatic merge from submit-queue
Cluster / InstanceGroup File Assets
@chrislovecnm @justinsb ...
The current implementation does not make it ease to fully customize nodes before kube install. This PR adds the ability to include file assets in the cluster and instaneGroup spec which can be consumed by nodeup. Allowing those whom need (i.e. me :-)) greater flexibilty around their nodes. @Note, nothing is enforced, so unless you've specified anything everything is as the same
- updated the cluster_spec.md to reflect the changes
- permit users to place inline files into the cluster and instance group specs
- added the ability to template the files, the Cluster and InstanceGroup specs are passed into context
- cleaned up and missed comment, unordered imports etc along the journey
notes: In addition to this; need to look at the detecting the changes in the cluster and instance group spec. Think out loud perhaps using a last_known_configuration annotation, similar to kubernetes
Automatic merge from submit-queue
Create cluster requirements for DigitalOcean
Initial changes required to create a cluster state. Running `kops update cluster --yes` does not work yet.
Note that DO has already adopted cloud controller managers (https://github.com/digitalocean/digitalocean-cloud-controller-manager) so we set `--cloud-provider=external`. This will end up being the case for aws, gce and vsphere over the next couple of releases.
https://github.com/kubernetes/kops/issues/2150
```bash
$ kops create cluster --cloud=digitalocean --name=dev.asykim.com --zones=tor1
I0821 18:47:06.302218 28623 create_cluster.go:845] Using SSH public key: /Users/AndrewSyKim/.ssh/id_rsa.pub
I0821 18:47:06.302293 28623 subnets.go:183] Assigned CIDR 172.20.32.0/19 to subnet tor1
Previewing changes that will be made:
I0821 18:47:11.457696 28623 executor.go:91] Tasks: 0 done / 27 total; 27 can run
I0821 18:47:12.113133 28623 executor.go:91] Tasks: 27 done / 27 total; 0 can run
Will create resources:
Keypair/kops
Subject o=system:masters,cn=kops
Type client
Keypair/kube-controller-manager
Subject cn=system:kube-controller-manager
Type client
Keypair/kube-proxy
Subject cn=system:kube-proxy
Type client
Keypair/kube-scheduler
Subject cn=system:kube-scheduler
Type client
Keypair/kubecfg
Subject o=system:masters,cn=kubecfg
Type client
Keypair/kubelet
Subject o=system:nodes,cn=kubelet
Type client
Keypair/kubelet-api
Subject cn=kubelet-api
Type client
Keypair/master
Subject cn=kubernetes-master
Type server
AlternateNames [100.64.0.1, 127.0.0.1, api.dev.asykim.com, api.internal.dev.asykim.com, kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local]
ManagedFile/dev.asykim.com-addons-bootstrap
Location addons/bootstrap-channel.yaml
ManagedFile/dev.asykim.com-addons-core.addons.k8s.io
Location addons/core.addons.k8s.io/v1.4.0.yaml
ManagedFile/dev.asykim.com-addons-dns-controller.addons.k8s.io-k8s-1.6
Location addons/dns-controller.addons.k8s.io/k8s-1.6.yaml
ManagedFile/dev.asykim.com-addons-dns-controller.addons.k8s.io-pre-k8s-1.6
Location addons/dns-controller.addons.k8s.io/pre-k8s-1.6.yaml
ManagedFile/dev.asykim.com-addons-kube-dns.addons.k8s.io-k8s-1.6
Location addons/kube-dns.addons.k8s.io/k8s-1.6.yaml
ManagedFile/dev.asykim.com-addons-kube-dns.addons.k8s.io-pre-k8s-1.6
Location addons/kube-dns.addons.k8s.io/pre-k8s-1.6.yaml
ManagedFile/dev.asykim.com-addons-limit-range.addons.k8s.io
Location addons/limit-range.addons.k8s.io/v1.5.0.yaml
ManagedFile/dev.asykim.com-addons-storage-aws.addons.k8s.io
Location addons/storage-aws.addons.k8s.io/v1.6.0.yaml
Secret/admin
Secret/kube
Secret/kube-proxy
Secret/kubelet
Secret/system:controller_manager
Secret/system:dns
Secret/system:logging
Secret/system:monitoring
Secret/system:scheduler
Must specify --yes to apply changes
Cluster configuration has been created.
Suggestions:
* list clusters with: kops get cluster
* edit this cluster with: kops edit cluster dev.asykim.com
* edit your node instance group: kops edit ig --name=dev.asykim.com nodes
* edit your master instance group: kops edit ig --name=dev.asykim.com master-tor1
Finally configure your cluster with: kops update cluster dev.asykim.com --yes
```
This enables external admission controller webhooks, api aggregation,
and anything else that relies on the
--proxy-client-cert-file/--proxy-client-key-file apiserver args.
Kashif Saadat