Automatic merge from submit-queue.
GCE Dump: Include instance IPs
The challenge here is that we normally only get the instance url. So we
have to do another call to GCE, but we also don't want to do one call per
instance.
Instead, we create a dump operation context object which we pass into the
dump.
Adding an extra option to the toolbox templating to format the YAML before writing out; which is usefull to cleanup formating issues and as detecting errors in the template
- added a formating options --format-yaml to the toolbox template which cleans up the yaml before writing out
- updated the cli documentation
- added the --config-value used to grab the configuration
The challenge here is that we normally only get the instance url. So we
have to do another call to GCE, but we also don't want to do one call
per instance.
Instead, we create a dump operation context object which we pass into the dump.
Automatic merge from submit-queue.
GCS: Use ACLs for GCE permissions
This needs less permissions, but also allows for more granular control over
access to files than whole-bucket permissions that IAM gives us.
This lets us configure cross-project permissions while ourselves needing
minimal permissions, but also gives us a nice hook for future lockdown
of object-level permissions.
Automatic merge from submit-queue.
Implement DigitalOcean Droplet FI Task
Implements cloudup fi tasks for DigitalOcean droplets. It makes a few assumptions to reduce the size of this PR, those will be addressed in future PRs.
Also does some cleanup in the DigitalOcean `dns` package.
The current implementation fails on template which reference unset variables, it is however useful at times to permit overriding this behavior and using sprig default() for example methods to handle it
- added a new command line option --fail-on-missing (defaults to true, so keeps the current behaviour)
- updated the unit test to reflect the changes
- updated the cli docs
Without this change, `kops get --name non-existing-cluster` will
exit with a status of `0`, which makes it difficult to check if
a cluster (and other resources) exist in a shell script.
Automatic merge from submit-queue.
bumping k8s versions in test files
updating k8s versions in test YAML files so that we do not get warnings during testing
Automatic merge from submit-queue.
Add Zones field to InstanceGroup
The Zones field can specify zones where they are not specified on a
Subnet, for example on GCE where we have regional subnets.
Automatic merge from submit-queue. .
promoting drain and validate by setting feature flag to true
I am unable to recreate https://github.com/kubernetes/kops/issues/2407, and frankly, it may be an edge case. We could warn a user if their wait times are low, but that would be another PR.
This PR moves Drain and Validate functionality for rolling-updates into the default user experience, setting the Feature Flag to true.
Per feedback, I am using the node and master interval times for the validation.
Automatic merge from submit-queue. .
Create GCE networks in auto mode, not legacy mode
auto mode allows for conversion to custom mode at the API level, and
legacy mode is deprecated.
- add a new kops secret command to store encryption config
- add the experimential-encryption-provider-config flag to the kube-apiserver
- add functionality to nodeup to create the stored encryption config in the correct path
Automatic merge from submit-queue
Add romana to built-in CNI options
This PR adds `romana` as a networking option for kops.
It installs the latest "preview" release of Romana v2.0, which provides the expected features in terms of IP allocations and route configuration. Network policy features are being ported to 2.0 and will be in the final release. (We intend to submit a followup PR for kops as part of that rolling out that release.)
Note: in this setup, we're using the etcd cluster that kops deploys for k8s. This isn't ideal, but some possibilities (eg: StatefulSets) aren't practical for the CNI itself, and creating a parallel etcd cluster via manifests seemed to be a more-intrusive approach than using the existing one.
If this is a concern or problem, then I'm very open to discussing and implementing it based on your suggestions.
Also, some functionality is exclusive to AWS environments. Other cloud platforms are on Romana's roadmap but not developed yet. Let me know that restriction needs to be enforced in code or directly documented.
Automatic merge from submit-queue
Flannel: change default backend type
We support udp, which has to the default for backwards-compatibility,
but also new clusters will now use vxlan.
Automatic merge from submit-queue
Toolbox template
Extending the current implementation of toolbox template to include multiple files and snippets. Note, I've removed the requirements for defaults as I think people should be forced to specifically pass them
- allowing the users to use a snippets directory for reusable templates
- allows the users to specify multiple templates files via multiple --template <path>, use a directory or both
- allows the users to specify multiple configuration files via multiple --values <path>, use a directory or both
- adding a safety check to ensure templates don't reference an unknown values
- fixing the vetting issues to the method YamlToJson -> YAMLToJSON
- as usual anything a saw on the journey which doesn't comply with go-vet got changed
Examples of a snippet
```YAML
hooks:
- name: some_service.service
manifest: |
{{ include "some_service.service" . | indent 6 }}
```
We currently use something similar to template our cluster and instances group documents, handling the differences between prod, ci and ephemeral
Extending the current implementation of toolbox template to include multiple files and snippets. Note, i've removed the requirements for defaults as I think people should be forced to specifically pass them.
- fixing the vetting iseues to the method YamlToJson -> YAMLToJSON
- adding a safety check to ensure templates don't reference an unknown value
- extending the unit test to ensure the above works on main and snippets
- include the ability to specify multiple configuration files, useful for common.yaml and prod.yaml etc
Requested Changes - Toolbox Templating
Added the requested changes
- moved the templater into it's own package rather than using base util
- moved to using the sprig library for additional template function
- @note: i couldn't find a native way in sprig to do snippets, also the i've overloaded the indent as it appears to do the indent on all lines rather than on the newline, meaning i'd have to shift my first line back by the indent to get it to work, which seems ugly
This will allow us to set CIDRs for nodeport access, which in turn will
allow e2e tests that require nodeport access to pass.
Then add a feature-flagged flag to `kops create cluster` to allow
arbitrary setting of spec values; currently the only value supported is
cluster.spec.nodePortAccess
Automatic merge from submit-queue
Implementing GCE as an interface - modelling aws cloud provider
GCE and other cloud providers are structs instead of an interface. AWS cloud provider implements an interface. This PR refactors `GCECloud` as an interface, and creates `gceCloudImplementation`.
- [x] Need to e2e test
Automatic merge from submit-queue
Kops command fixes
TrimSuffix() on defaultStateStore doesn't actually do anything, and there is an unused answer variable in the tests.
Automatic merge from submit-queue
Allow the strict IAM policies to be optional
The stricter IAM policies could potentially cause regression for some edge-cases, or may rely on nodeup image changes that haven't yet been deployed / tagged officially (currently the case on master branch since PR https://github.com/kubernetes/kops/pull/3158 was merged in).
This PR just wraps the new IAM policy rules around a cluster spec flag, `EnableStrictIAM`, so will default to the original behaviour (where the S3 policies were completely open). Could also be used to wrap PR https://github.com/kubernetes/kops/pull/3186 if it progresses any further.
- Or we could reject this and have the policies always strict! :)
Automatic merge from submit-queue
starting work on file assets builder
I refactored to the dockerassets pkg to assetstasks, in order to not add yet another package. Added file copy task, that I have tested with s3 locally, but not certain how to add memfs tests.
Fixes: https://github.com/kubernetes/kops/issues/3086
Automatic merge from submit-queue
Add cluster spec to node user data so component config changes are detected
Related to #3076
Some cluster changes such as component config modifications are not picked up when performing updates (nodes are not marked as `NEEDUPDATE`). This change introduces the ability to:
1. Include certain cluster specs within the node user data file ~(`enableClusterSpecInUserData: true`)~
2. ~Encode the cluster spec string before placing within the user data file (`enableClusterSpecInUserData: true`)~
~The above flags default to false so shouldn't cause any changes to existing clusters.~
Following feedback I've removed the optional API flags, so component config is included by default within the user data. This WILL cause all nodes to have a required update to their bootstrap scripts.
Automatic merge from submit-queue
Clarify docs: rename spec/specification into desired configuration
The cluster state in S3 has (among others) two files: `cluster.spec` and `config`.
When the documentation mentioned "create or update cluster spec" for example, it was confusing what was actually updated. It's not the cluster.spec file.
As I understand, `cluster.spec` should only be created/updated after `kops update --yes` is run.
I changed the docs for `kops get`, `kops create`, `kops replace`, `kops edit`.
I did NOT change those files: `kops_rolling-update.md`, `kops_rolling-update_cluster.md` as I think those actually use `cluster.spec`.
In the S3 bucket, the file cluster.spec is not actually the spec, but the
actual configuration. The file config is the spec. To avoid confusion,
this commit changes spec/specification into 'desired configuration' in
the documentation, to avoid associating cluster.spec with a cluster
'specification' that the users should use.
The current 'kops replace' fails if the resource does not exist, which is annoying if you want to use the feature to drive your CI. This PR adds a --create option to create any resource which does not exist. At the moment we limit this to instanceGroups only. I'd also like to see this command perhaps be renamed to kops apply?
Automatic merge from submit-queue
adding warning about --full
This starts work on https://github.com/kubernetes/kops/issues/2290. Adds a warning to deter users from using YAML generated by --full.
This adds a well-known secret name `nodedockercfg` which will automatically
be used if present to create /root/.docker/config.json on all nodes. This will
allow private registries to be used for kops hooks as well as any k8s images
without the need to define `imagePullSecrets` in every namespace.
closes https://github.com/kubernetes/kops/issues/2505
We expose the Shared state on the resource, bubble it down into derived
resources (like NAT Gateways or Internet Gateways).
Deletion then skips Shared items. Listing (currently) does not skip
them.
* Use the pretty.LongDesc function (which doesn't pipe it through
markdown -> ascii)
* Use pretty.Bash helper to avoid problems with backticks in
go-backtick-quoted-strings
* Make sure indentation is consistent
We modelled our VFS clientset (for API objects backed by a VFS path)
after the "real" clientsets, so now it is relatively easy to add a
second implementation that will be backed by a real clientset.
The snafu here is that we weren't really using namespaces previously.
Namespaces do seem to be the primary RBAC scoping mechanism though, so
we start using them with the real clientset.
The namespace is currently inferred from the cluster name. We map dots
to dashes, because of namespace limitations, which could yield
collisions, but we'll deal with this by simply preventing users from
creating conflicting cluster names - i.e. you simply won't be able to
create a.b.example.com and a-b.example.com
Kubernetes Submit Queue