kubernetes
/
kops
mirror of https://github.com/kubernetes/kops.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
5,172 Commits 31 Branches 256 Tags 544 MiB
Commit Graph

1453 Commits

Author SHA1 Message Date
Justin Santa Barbara 310ba12edf Fix log message of shell commands 
We were double-printing argv[0]
 2016-07-29 01:30:34 -04:00
Justin Santa Barbara 19e19a0f09 Add `kops get secrets` command 
Matches our naming pattern, and now using our text grid output
 2016-07-29 01:12:03 -04:00
Justin Santa Barbara 3e3ca5a2b8 Merge pull request #222 from justinsb/fix_204 
Run the master on the pod network, unless IsolateMaster=true
 2016-07-29 00:51:50 -04:00
Justin Santa Barbara bd3ab166b7 Import SSH public key into the keystore 
This means it only needs to be specified during `kops create`.  We
remove the option from `kops update` for consistency.

This will shortly be manageable using the secrets functionality.

Fix #221
 2016-07-29 00:45:37 -04:00
Justin Santa Barbara 2b3f55563e Run the master on the pod network, unless IsolateMaster=true 
The master is now registered as a Node.  It is marked as Unschedulable,
so normal pods will not run on it.  But Daemonsets will, and it is
surprising that they don't work unless hostNetwork=true.

The default is now what seems to be expected:
* we allocate the master a real CIDR on the pod network
* kube-proxy runs on the master, so it can talk to pods
* we run kubelet on the master with enable-debugging-handlers, so
  kubectl logs etc works

To get the old behaviour, edit the cluster spec and set
`isolateMasters: true`
 2016-07-28 12:12:16 -04:00
Justin Santa Barbara e3e58f605e support multiple zones in `kops import` 
Seems to be less special-cased also.

Fix #216
 2016-07-27 11:03:22 -04:00
Justin Santa Barbara 97cb6a7296 Add test for bigint format 2016-07-24 21:49:35 -04:00
Justin Santa Barbara 9a741043bc Fix build with golang 1.5 
Replace big.Int Text(10) with String()

Also create a makefile target to test building with golang 1.5:
check-builds-in-go15

Fixes #163
 2016-07-22 22:49:54 -04:00
Justin Santa Barbara 1240e1d924 Fix bug where edit cluster was computing with the old spec 
Also add some tests, and fix some variable naming to hopefully stop it
happening again.

Fix #194
 2016-07-22 18:19:40 -04:00
Justin Santa Barbara bbf1755f57 Discover an existing InternetGateway when running with Terraform Output 
We currently don't do discovery of existing resources when running with
Terraform output, but when using a shared VPC this is wrong - we need to
reuse the existing gateway.

Directly discover the existing gateway.  We might need to do full
discovery later, but for now this is sufficient.

Fix #181
 2016-07-22 12:48:11 -04:00
Justin Santa Barbara 9e9855d1a4 Simpler upgrade procedure: reuse subnet 
By reusing the subnet & security groups, we are able to skip the ELB
steps of the upgrade procedure.  The new cluster also has the same
identity as the old cluster for security groups, so we don't need to
reconfigure ELB etc.

Fixes #175
Fixes #174
 2016-07-22 11:47:12 -04:00
Justin Santa Barbara 31892fb08d Refactor tests for cleaner separation 2016-07-22 10:19:01 -04:00
Justin Santa Barbara 11d51b04a9 Adapt IAM policies when running in cn-north-1 
Fix #27
 2016-07-21 22:19:43 -04:00
Justin Santa Barbara 2fa3bcc952 UX: Split create command into `create` and `update` 
We separate out the `create cluster` operation from the `update cluster`
operation.  Now create cluster only creates the spec (unless you pass
--yes), and is only for new clusters.

`update cluster` works on new or existing clusters, and should be called
to apply changes.

`update` is not the best name, because it means something different in
kubectl, but I think it's a good start.
 2016-07-21 11:54:09 -04:00
Justin Santa Barbara c0477381c1 Fixes for terraform block device mapping output 2016-07-21 10:09:16 -04:00
Justin Santa Barbara 8fdda787b9 Merge pull request #154 from justinsb/fix_153 
Populate kube-apiserver apiserver-count flag for HA
 2016-07-21 09:30:45 -04:00
Justin Santa Barbara 50159fe47d Cleaner dryrun output 2016-07-21 03:25:57 -04:00
Justin Santa Barbara f67aea54d9 Guard against zero values when comparing values 
This seems to happen when comparing map values
 2016-07-20 19:55:59 -04:00
Justin Santa Barbara ba4b0e303b Fix nil dereference correctly 
(I hope)

Also cover more of the cases

Fix #167
 2016-07-18 15:43:06 -04:00
Justin Santa Barbara 63806c315e Avoid segfault on nil ResourceHolder 2016-07-18 15:35:19 -04:00
Justin Santa Barbara fed46864c7 Populate kube-apiserver apiserver-count flag for HA 
Fix #153
 2016-07-14 23:35:41 -04:00
Justin Santa Barbara 18af54de6e Fix missing string cast 2016-07-14 11:06:39 -04:00
Justin Santa Barbara 47ef67a71d Merge pull request #147 from justinsb/fix_69 
Clearer error message when importing a cluster with a custom subnet
 2016-07-14 11:05:33 -04:00
Justin Santa Barbara 3db3096c1b Merge pull request #149 from justinsb/fix_148 
Import the node instance type when importing a cluster
 2016-07-14 11:04:29 -04:00
Justin Santa Barbara 57d6e0d8d9 Import the node instance type when importing a cluster 
We now query for the launchconfiguration and import the first machine
type we find.

Fix #148
 2016-07-14 10:51:39 -04:00
Justin Santa Barbara 7c1177cc82 Clearer error message when importing a cluster with a custom subnet 
We can at least give the user better instructions on how to proceed.
Because this is not in general safe though, I'm not sure we want to
gloss over this.

Issue #69
 2016-07-14 10:38:32 -04:00
Justin Santa Barbara 302f23463e Configuration of admin access to ports 22 and master-443 
Fix #143
 2016-07-14 10:33:26 -04:00
Justin Santa Barbara 1102e7818d Perform as much validation as possible before writing a cluster config 
This prevents us getting into a situation where an invalid cluster is
saved, and then cannot be deleted.
 2016-07-14 09:54:21 -04:00
Justin Santa Barbara 1a5a917c9d Hash the k8s assets 
This serves both as a security check, but also validates early that the
assets actually exist.

Fix #98
 2016-07-14 09:46:48 -04:00
Justin Santa Barbara df56204479 Export kubecfg after as create cluster by default 
It is scoped to a particular context, so seems harmless, and users will
(almost?) always do it after creation.

Fix #129
 2016-07-12 23:02:43 -04:00
Justin Santa Barbara fb1fa97bec Fix DNS zone creation 
Though we don't actually expect to use it, we now pass the nonce that
was missing.

Fix #29
 2016-07-12 22:16:48 -04:00
Justin Santa Barbara cae256340b Find matching existing DNS hosted zone 
We choose the longest matching existing hosted zone as the default,
rather than the previous heuristic which was overly simplistic.

We also require the hosted zone to exist now; it doesn't seem to really
cost us anything given that the user has to set up DNS delegation
anyway.

Fix #125
 2016-07-12 22:15:20 -04:00
Justin Santa Barbara b69ee2a0ee Accept a hosted zone ID for dns-zone 
There isn't any real ambiguity.

Fix #75
 2016-07-12 22:14:26 -04:00
Justin Santa Barbara d1b399a97d Fix reissue of CA key 2016-07-11 00:09:47 -04:00
Justin Santa Barbara 0f4999aade Fix cluster validation so it is less strict 
We now validate the user-provided spec, so we can't enforce that all
fields are populated.  Instead we enforce that if they are populated
they are not invalid.
 2016-07-11 00:09:12 -04:00
Justin Santa Barbara 0aed68c260 Add support for deleting an InstanceGroup 2016-07-11 00:08:55 -04:00
Justin Santa Barbara c4f2fbfcaf Tweaks to rolling-update CLI 2016-07-11 00:08:21 -04:00
Justin Santa Barbara 1d59f2aa80 Replace StateStore with a registry 
StateStore was highly orientated towards a VFS system; replace it with a
Registry abstraction that is more object based.

We also rationalize much of the CLI (cmd) command logic also.
 2016-07-11 00:07:59 -04:00
Justin Santa Barbara eb589fd025 Import cluster should create the same style name 2016-07-11 00:03:22 -04:00
Justin Santa Barbara f771c2af4c Add support for spot instances 
Fixes #58
 2016-07-10 23:56:16 -04:00
Justin Santa Barbara 4ac0395a9f Merge pull request #103 from justinsb/delete_shutting_down_state 
Tolerate shutting-down state on instance during delete
 2016-07-09 22:32:21 -04:00
Justin Santa Barbara 0c867a8971 Merge pull request #108 from justinsb/fix_96 
Detect & delete new ASG launch configs
 2016-07-09 22:29:56 -04:00
Justin Santa Barbara 524fb645d0 Delete cluster deletes keypairs 
Again we delete by name, which is why we name them by the very unlikely
to conflict name "kubernetes.<clustername>"

Issue #107
 2016-07-09 22:26:12 -04:00
Justin Santa Barbara 5b8b4d4da3 Detect & delete new ASG launch configs 
We now output a ClusterName property into the launchconfig, even though
we don't technically need it.  But it allows us to more easily detect
the cluster, and it generally seems like a good idea.

Also rename to 'autoscaling-config' and clean up the cluster name
detection logic.

Fix #96
 2016-07-09 22:07:24 -04:00
Justin Santa Barbara b98422fb3a Merge pull request #101 from justinsb/eventual_consistency_igw_tag 
Tolerate eventual consistency error tagging gateways
 2016-07-09 22:03:40 -04:00
Justin Santa Barbara d705a1001c Clean up IAM roles 
We match by name; this seems acceptable given that we make the cluster
name a FQDN.

Issue #97
 2016-07-09 15:33:57 -04:00
Justin Santa Barbara fdfe710c92 Merge pull request #104 from justinsb/fix_35 
More tweaks to the IAM async creation tolerance
 2016-07-09 13:49:15 -04:00
Justin Santa Barbara 0a8b94e748 More tweaks to the IAM async creation tolerance 
Issue #35
 2016-07-09 02:45:30 -04:00
Justin Santa Barbara c058a0c2a8 Tolerate shutting-down state on instance during delete 2016-07-09 02:41:56 -04:00
Justin Santa Barbara 5c65600229 Delete Route53 host records on cluster-down 
Initial implementation: we match by name, for example; when we have a
hosted zone id we should plumb it through.

Does implement bulk deletion, because on Route53 we have to delete
multiple resources at once.

Issue #74
 2016-07-09 02:41:25 -04:00
Justin Santa Barbara 935b08fba2 Tolerate eventual consistency error tagging gateways 2016-07-09 01:46:32 -04:00
Justin Santa Barbara 6c6410dc81 Merge pull request #87 from justinsb/fix_35_part2 
Report less-scary message when IAM instance profile is not ready yet
 2016-07-09 01:25:28 -04:00
Justin Santa Barbara 13b8e81bd6 Allow configurable RootDeviceSize & RootDeviceType 
This allows for a larger EBS root volume (and we now default to 20GB,
just like kube-up did).

We remove the BlockDeviceMappings support because it wasn't used and
made things a lot more complicated.  We always map the ephemeral
devices.

Issue #24
 2016-07-08 01:11:14 -04:00
Justin Santa Barbara d1ef7580a9 Report less-scary message when IAM instance profile is not ready yet 
IAM instance profile creation is very async, and this causes dependent
resources to fail.  That's fine - we have good retry logic - but we
should output a less frightening error message.

Issue #35
 2016-07-07 10:22:44 -04:00
Justin Santa Barbara 4bf51a825b More info on "clusters cannot span multiple regions" 2016-07-07 09:49:23 -04:00
Justin Santa Barbara a6ba62abe9 More instance type support: g2, i2, x1 families 2016-07-06 23:27:08 -04:00
Justin Santa Barbara f89c80c52c Merge pull request #78 from justinsb/fix_restart_1 
Make sure that we enable systemd services
 2016-07-06 23:25:07 -04:00
Justin Santa Barbara 09dd3002ff Make sure that we enable systemd services 
We were previously just starting them, which works fine for the new
machine and machine termination case, but doesn't work when doing a
machine restart (e.g. init 6)

Issue #70
 2016-07-06 23:04:51 -04:00
Justin Santa Barbara 40cbdc4d00 Add support for r3 family 
Fix #76
 2016-07-06 22:02:56 -04:00
Justin Santa Barbara fcd644441e Merge pull request #61 from justinsb/typo_1 
Fix typo in error message
 2016-07-06 16:42:47 -04:00
Justin Santa Barbara 957d30a157 Merge pull request #68 from justinsb/fix_elb_pagesize 
Fetch ELB tags in chunks of 20
 2016-07-06 13:47:30 -04:00
Justin Santa Barbara 274a8eff3c Fetch ELB tags in chunks of 20 
ELB DescribeTags has a limit of 20 ELBs / call.  So we paginate the
DescribeLoadBalancers call with page size = 20 also.
 2016-07-06 13:46:16 -04:00
Justin Santa Barbara 7347f1815a Parse SSH public keys with 'odd' whitespace 
e.g. a key with \r instead of spaces

Fix #63
 2016-07-06 11:18:09 -04:00
Justin Santa Barbara a91cc6e9a4 Add tests for SSH key fingerprint 
Issue #63
 2016-07-06 10:47:20 -04:00
Justin Santa Barbara 4dc869756f Quote public key in error message 
Just in case it contains invalid characters

Issue #63
 2016-07-06 10:22:09 -04:00
Justin Santa Barbara b6035bffaf Fix typo in error message 
Fingerpring -> Fingerprint
 2016-07-06 10:04:13 -04:00
Justin Santa Barbara db4b190831 Fix panic when IAMInstanceProfile not found 2016-07-06 09:51:44 -04:00
Justin Santa Barbara e953d76f9f Merge pull request #51 from justinsb/fix_48 
Only warn if a _different_ ephemeral device is attached
 2016-07-06 03:07:05 -04:00
Justin Santa Barbara fa3157e512 Merge pull request #54 from justinsb/fix_tests 
Update & get tests working again
 2016-07-06 03:06:59 -04:00
Justin Santa Barbara 981cbe0189 Merge pull request #55 from justinsb/fix_22 
Fix #22
 2016-07-06 03:06:45 -04:00
Justin Santa Barbara 6f49a50886 Wait for InstanceProfile to be readable after creation 
This should avoid spurious errors when creating dependent resources,
which fail if we try to create before the InstanceProfile is fully
created.

Issue #35
 2016-07-06 01:01:24 -04:00
Justin Santa Barbara 2f84573e2b DebugPrint should handle the case where the resource cannot be found 
We print "unknown resource" instead

Issue #22
 2016-07-06 00:40:51 -04:00
Justin Santa Barbara 7b57329f34 Don't panic if asked to Open an empty resource 
Return an error instead

Issue #22
 2016-07-06 00:37:18 -04:00
Justin Santa Barbara c956bdada5 Update & get tests working again 
They're still hitting the AWS APIs, have terrible coverage and are slow.
But it's a start!
 2016-07-06 00:33:07 -04:00
Justin Santa Barbara 05af11b72e Only warn if a _different_ ephemeral device is attached 
When we retry a task, we run the Run method again.  But in this case,
the run method actually populated some default values.  Only warn if the
values we are populating are different, to avoid spurious warnings.

We probably need a stronger lifecycle - for example having a Validate
method would probably be helpful.

Fix #48
 2016-07-05 22:05:42 -04:00
Justin Santa Barbara 9c2f92f289 AWS: set hostname-override from metadata service 
This is a weird edge case, because it can't be determined in advance.

We carve out a special well-known name, `@aws`, which nodeup/protokube
will expand to the local-hostname from the aws metadata service when it
is found in the HostnameOverride value.

Ideally we wouldn't do this at all now that we have DNS integration, but
we first want to get into the tested & working configuration!

Fixes #19
 2016-07-05 11:36:47 -04:00
Justin Santa Barbara 8e4d751eb9 Update schema to sync up with coming changes in API 
Thankfully we aren't yet using any of these properties
 2016-07-02 14:58:34 -04:00
Justin Santa Barbara 0fbb35dec0 Merge pull request #10 from justinsb/az_diagnostics 
More diagnostics when an AZ is not recognized
 2016-07-01 10:07:12 -04:00
Justin Santa Barbara 58fbef0572 Apply gofmt 2016-07-01 10:02:30 -04:00
Justin Santa Barbara bfdd976855 More diagnostics when an EC2 zone is not recognized 2016-07-01 10:00:19 -04:00
Justin Santa Barbara ac119289fd Add ap-south-1 region 2016-07-01 09:59:56 -04:00
Justin Santa Barbara d8b661777b Merge cloudup tool into `kops create cluster` 2016-06-30 10:25:50 -04:00
Justin Santa Barbara d427858477 Rename to kops 
The upup tool is now called kops, and we have moved repos
 2016-06-30 09:25:25 -04:00
Justin Santa Barbara be4be57f3a upup: workaround for secrets problem with upgrade 2016-06-28 13:08:41 -04:00
Justin Santa Barbara 80c1e40784 upup: Add get commands 2016-06-28 11:49:41 -04:00
Justin Santa Barbara e3062a9f51 upup: improved upgrade procedure 2016-06-28 11:49:17 -04:00
Justin Santa Barbara e315564cfa upup: More delete cleanup, use statestore by default 2016-06-28 11:47:19 -04:00
Justin Santa Barbara cfcc2f1ba9 Make StateStore a top-level concept, with subdirs for each cluster 2016-06-27 16:21:51 -04:00
Justin Santa Barbara 9ff5dcd902 Make IAM permission abstraction more generic 
At least in the JSON representation!
 2016-06-27 15:48:16 -04:00
Justin Santa Barbara 947a045667 Rename DNSDomain -> ClusterDNSDomain for clarity 2016-06-27 15:36:11 -04:00
Justin Santa Barbara 26d05341b4 Move options to common stage, so that it works with terraform generation 2016-06-27 15:21:31 -04:00
Justin Santa Barbara c36607644b Better shared VPC support: more validation 2016-06-27 15:00:51 -04:00
Justin Santa Barbara ee8bdb907e upup delete: don't always print dependencies 2016-06-27 15:00:51 -04:00
Justin Santa Barbara 65135db3be upup: group DNS properties 2016-06-27 15:00:51 -04:00
Justin Santa Barbara b6cf38c96e AllocateNodeCIDRs need no longer be "bubbled down" 
We have it on the KCM config; just set it there
 2016-06-27 00:32:19 -04:00
Justin Santa Barbara 6932ff4a06 Make sure terraform output is stable across runs 
Fixes #132
 2016-06-27 00:04:43 -04:00
Justin Santa Barbara 87b7c71fcc Remove spurious errors when deleting a cluster 2016-06-27 00:03:49 -04:00
Justin Santa Barbara 492856069a Apply gofmt 2016-06-26 23:24:17 -04:00
Justin Santa Barbara eeed4a3031 Rationalize API to something we want to support forever 2016-06-26 23:09:02 -04:00
Justin Santa Barbara ee325435e6 Rationalize properties to the minimal set 2016-06-26 09:45:05 -04:00
Justin Santa Barbara 0a6260f7c4 Merge pull request #130 from justinsb/upup_vfs_context 
upup: Add VFS context object, centralize usage
 2016-06-23 10:30:09 -04:00
Justin Santa Barbara 3185a3fe5c Fix upgrade 2016-06-23 10:28:17 -04:00
Justin Santa Barbara 75af0d089d upup: Add VFS context object, centralize usage 
Replaces ad-hoc construction of VFS paths
 2016-06-23 10:27:32 -04:00
Justin Santa Barbara 506d0d555a Merge pull request #127 from justinsb/upup_upgrade_tooling 
upup: improved delete; create upup export and upup upgrade
 2016-06-23 10:27:32 -04:00
Justin Santa Barbara 90d7fb87ad upup: improved delete; create upup export and upup upgrade 2016-06-23 08:59:20 -04:00
Justin Santa Barbara 93f634b428 upup: use vfs for secretstore/keystore 
This is needed so that we can have encrypted storage and complex keys
(e.g. multiple CA certs).  Multiple CA certs are needed for an in-place
upgrade from kube-up v1.
 2016-06-23 08:58:54 -04:00
Justin Santa Barbara 75f386235a upup: rolling-update add mutex lock 2016-06-17 09:36:43 -04:00
Justin Santa Barbara 1e7159a923 upup: detach internet gateway if vpc is to be deleted 2016-06-17 09:36:27 -04:00
Justin Santa Barbara 3ede6c1f4a upup: uncompress gzipped data in delete cluster 2016-06-17 09:36:12 -04:00
Justin Santa Barbara 3040676a3b upup: enable subnet changes on ASG 
For kube-up upgrade
 2016-06-15 22:56:19 -04:00
Justin Santa Barbara 22bf484777 upup: Assign subnet CIDRs based on AZs 2016-06-15 14:03:19 -04:00
Justin Santa Barbara b147091e5c upup: fix dryrun when adding a subnet 
When adding a subnet in --dryrun, the ID might not be set during a Find
 2016-06-15 14:03:19 -04:00
Justin Santa Barbara a10c76a954 upup: prevent spurious comparison failure on VPC/IGW 2016-06-15 14:03:19 -04:00
Justin Santa Barbara eb23e22b3c upup: rolling-update should take --name, not --cluster-id 2016-06-13 17:37:30 -04:00
Justin Santa Barbara 14bae04e01 upup: turn down a few more logging messages 2016-06-13 17:37:30 -04:00
Justin Santa Barbara 0559ec1210 upup: Support for shared VPCs 
A lot of work that had to happen here:

* Better reuse of config
* Ability to mark VPC & InternetGateway as shared
* Find models relative to the executable, to run from a dir-per-cluster

Fixes #95
 2016-06-13 11:37:06 -04:00
Justin Santa Barbara 530b4c79db upup: experimental addon management 2016-06-13 11:35:00 -04:00
Justin Santa Barbara 55574ec3f5 upup: apply gofmt 2016-06-11 23:09:44 -04:00
Justin Santa Barbara 226f0284b1 upup: Fix log message 2016-06-11 23:09:44 -04:00
Justin Santa Barbara 72afa859a3 upup: delete cluster should take -name to be consistent 
Still not entirely sure on cluster-id vs name, but name is consistent
 2016-06-11 23:08:40 -04:00
Justin Santa Barbara 96247daf6d Merge pull request #101 from justinsb/upup_fix_delete_asg 
upup: fixes for ASG deletion
 2016-06-11 23:06:00 -04:00
Justin Santa Barbara f9ac7af7df Merge pull request #100 from justinsb/upup_zones 
upup: separate node & master zone configuration; validate
 2016-06-11 23:05:55 -04:00
Justin Santa Barbara ee056fa975 upup: Retry if race detected in FindOrCreateSecret 
We do a read / create-if-empty.  If the create fails because of a
concurrent creation, we retry the operation.

Fix #73
 2016-06-11 22:40:47 -04:00
Justin Santa Barbara 0263826163 upup: delete cluster should check status of ASG 
Before deleting the ASG, we need to check it exists.

Issue #99
 2016-06-11 21:33:56 -04:00
Justin Santa Barbara e63426ce65 upup: delete cluster should eventually give up 
If it isn't making progress, eventually we should bail out so that we
can use this in scripts.  However, we don't need to be too aggressive,
because it is not good to leak resources, and a user will likely
Control-C us pretty fast when they see they're in a loop.

Issue #99
 2016-06-11 21:33:56 -04:00
Justin Santa Barbara b52877e2ce upup: separate node & master zone configuration; validate 
We allow --zones & --master-zones to be specified separately now, but we
validate for common errors (using a region where you meant a zone,
duplicating a zone, spanning regions, entering an invalid AZ etc)
 2016-06-11 21:06:31 -04:00
Justin Santa Barbara d489fccd28 upup: make NodeConfig be JSON-clean 
We use JSON-merging in our option parsing, so we need it to only
populate values that are actually set.
 2016-06-10 13:37:48 -04:00
Justin Santa Barbara f3ebc088f1 upup: make LaunchConfiguration compatible with terraform 
This way we can output a LaunchConfiguration prefix into terrform that
we can then read later, so that we can create with terraform and then
transfer to another mode of operation if desired.
 2016-06-10 13:36:36 -04:00
Justin Santa Barbara c54ef2773d upup: support Base64 and raw-string JSON encoding of certs/keys 
For compatability reasons, we write the certificate & keys as base64
encoded strings.  I don't think we have to any more, but we have to be
able to parse it.
 2016-06-10 13:35:40 -04:00
Justin Santa Barbara 7543d1c19d upup: apply gofmt 
Missed a few gofmt places
 2016-06-10 11:41:10 -04:00
Justin Santa Barbara 1cbcb82fe9 upup: use JSON as fallback in --dryrun printing 
Rather than giving up with an error, we can simply use JSON format.
 2016-06-10 11:38:21 -04:00
Justin Santa Barbara 9b32edfb81 upup: add experimental cluster rolling-update command 
Little more than a hack right now, but a good place to start.
 2016-06-10 11:36:57 -04:00
Justin Santa Barbara 52496ac73a upup: split launchconfiguration from ASG 
It is much more logical this way, and mirrors the way GCE & terraform
work.
 2016-06-10 11:36:17 -04:00
Justin Santa Barbara eeabcc0cb1 upup: fix options processing 
We want to make sure that any values we set are treated both as defaults
and as overrides (i.e. the options shouldn't be able to override a value
the user has explicitly set)
 2016-06-10 11:35:31 -04:00
Justin Santa Barbara 6e7c4209de upup: Simple tag fixes to avoid comparison failures 
We populate the actual expected tags in-place now
 2016-06-10 11:33:33 -04:00
Justin Santa Barbara 2c5e0c0164 upup: populate Name, to support --dryrun 
Some of these types were coming up as spurious changes.
 2016-06-10 11:33:29 -04:00
Justin Santa Barbara e10f09a81f Merge pull request #83 from justinsb/upup_node_sizes 
upup: node size on CLI and more instance type families
 2016-06-10 11:32:49 -04:00
Justin Santa Barbara 2bacc2035c upup: define m3, m4, c4 families 
This allows us to set up their ephemeral volumes
 2016-06-10 11:31:20 -04:00
Justin Santa Barbara 7c478c794f upup: harmonize model with official config types 2016-06-10 01:28:44 -04:00
Justin Santa Barbara 6e203da852 upup: split model into two parts 
This is probably a good idea anyway, but it also lets us side-step the
terraform no-dots-in-tags bug.
 2016-06-09 23:14:36 -04:00
Justin Santa Barbara fa860ca730 Merge pull request #75 from justinsb/upup_vfs 
upup: Add VFS for storing state in S3 or locally
 2016-06-09 23:12:54 -04:00
Justin Santa Barbara 338f158d22 upup: Add VFS for storing state in S3 or locally 
This also fixes a few concurrency issues, because we're writing in one
place now.
 2016-06-09 23:08:10 -04:00
Justin Santa Barbara 93284d5d13 upup: reformat terraform output 
Using string manipulation, sadly, but I believe it's safe as we don't
use heredocs
 2016-06-09 23:07:12 -04:00
Justin Santa Barbara c826f46a60 upup: support for terraform on AWS 
All seems good except for a bug with volume tagging
 2016-06-08 12:19:15 -04:00
Justin Santa Barbara 70d7c1b92e upup: avoid warning when building filters with nil name 2016-06-08 12:18:07 -04:00
Justin Santa Barbara 42e32f7379 upup: include kope-routing, but only if _kope_routing is set 2016-06-08 12:18:04 -04:00
Justin Santa Barbara 59124fd6fb upup: include count of tasks as we are executing them 2016-06-08 12:16:19 -04:00
Justin Santa Barbara 4d331e2881 upup: Reattempt up to 3 times when no progress made 
Now that we're running in parallel, sometimes AWS eventual consistency
causes us problems.  We now retry up to 3 times, sleeping 10 seconds in
between each run even when we aren't making progress.
 2016-06-08 12:16:19 -04:00
Justin Santa Barbara 44507e64cf upup: remove some bad logging 
Was left over from move to simpler unmarshal model
 2016-06-08 12:16:19 -04:00
Justin Santa Barbara 1eaf0d36a8 upup: HA support 
Specifying multiple zones will bring up an HA cluster.
 2016-06-07 15:44:00 -04:00
Justin Santa Barbara 6ea399fc79 upup: Add back in CompareWithID functions 
Removed when we pulled this functionality out of the go generator, but
then the manual code not re-added.

Also a few other small tweaks to Tasks.
 2016-06-07 15:37:07 -04:00
Justin Santa Barbara 4c0f54f95d upup: run create & delete in parallel 
It makes logging harder, but we want to start testing it!
 2016-06-07 15:23:07 -04:00
Justin Santa Barbara 6cf5cd423e upup: apply IAM changes 
We now apply changes to IAM policies, and print the diffs.
 2016-06-07 15:17:59 -04:00
Justin Santa Barbara c95656177f cloudup: Add support for ELB health checks 2016-06-07 09:01:35 -04:00
Justin Santa Barbara 062d2ece96 Merge pull request #54 from justinsb/upup_protokube 
Use protokube with upup
 2016-06-07 08:56:23 -04:00
Justin Santa Barbara 274696c0c5 Merge pull request #58 from justinsb/upup_smarter_delete 
upup: better cluster delete
 2016-06-07 08:47:21 -04:00
Justin Santa Barbara cf840ef21a upup: better cluster delete 2016-06-07 08:46:28 -04:00
Justin Santa Barbara e18833fcb7 Merge pull request #57 from justinsb/upup_resource_tagging 
cloudup: better retrying around resource tagging
 2016-06-07 08:43:21 -04:00
Justin Santa Barbara 9d6847ff49 cloudup: better retrying around resource tagging 2016-06-04 16:14:00 -04:00
Justin Santa Barbara 82f373cba0 nodeup: make sure tasks implement HasDependencies 2016-06-04 16:13:25 -04:00
Justin Santa Barbara 540c46fe4e nodeup: pass --master argument to protokube 2016-06-04 16:11:11 -04:00
Justin Santa Barbara c774777160 Merge pull request #50 from justinsb/upup_keys 
upup: better secrets support
 2016-06-04 15:57:08 -04:00
Justin Santa Barbara 4a85ef64fb Merge pull request #48 from justinsb/upup_elb 
upup: simple ELB support
 2016-06-04 15:56:57 -04:00
Justin Santa Barbara b9597c53ec Merge pull request #47 from justinsb/upup_delete_cluster 
upup: simple command to delete (AWS) clusters
 2016-06-04 15:56:51 -04:00
Justin Santa Barbara 5f4c03cd11 Merge pull request #46 from justinsb/upup_route53 
upup: Simple route53 (DNS) support
 2016-06-04 15:56:48 -04:00
Justin Santa Barbara a4408f76be upup: better secrets support 
Start creating commands to manage secrets, and also stop implicitly
creating them.
 2016-05-30 18:47:20 -04:00
Justin Santa Barbara 095bf95fc9 upup: simple command to delete (AWS) clusters 
A relatively dumb retry strategy to work around dependencies, but it
works and it is difficult to do _much_ better.
 2016-05-30 17:57:40 -04:00
Justin Santa Barbara 3a660fe8fa upup: simple ELB support 
Add support for ELB, though this doesn't wire it up yet
 2016-05-30 17:53:05 -04:00
Justin Santa Barbara 3389323392 upup: Simple route53 (DNS) support 2016-05-30 17:50:55 -04:00
Justin Santa Barbara 3aeaf73788 upup: move string-slice comparison functions to utils 2016-05-30 17:46:36 -04:00
Justin Santa Barbara 1aeea67510 upup: replace parse code with standard json unmarshaller 
Instead of reimplementing the unmarshal code, we implement a trick: we
implement an alternative JSON representation of an object: a string.

We unmarshal as normal, and then we reconcile these pointer values to
the primary values, by walking the unmarshalled tree.
 2016-05-30 17:46:36 -04:00
Justin Santa Barbara c72593fcf7 Use generators in tasks 2016-05-30 17:46:36 -04:00
Justin Santa Barbara 16559c9010 upup: Executor that performs some retry logic 
If there is an error performing a task, we will reattempt it as long as
forward progress is still being made (i.e. at least one other task
completed successfully)

This makes everything more reliable (though we should still fix these
problems), but it also lays the groundwork for parallel execution.
 2016-05-22 21:02:53 +01:00
Justin Santa Barbara 505ef056d9 Add upup command, with a subcommand to generate kubecfg 2016-05-17 12:54:13 -04:00
Justin Santa Barbara bbc460f5a7 Refactor Certificate and PrivateKey to implement io.WriterTo 2016-05-17 12:54:13 -04:00
Mike Danese 19bb767767 Merge pull request #28 from justinsb/upup_aws_retry_tags_on_eventual_consistency_error 
upup: tolerate aws eventual-consistency errors on tags
 2016-05-17 09:34:53 -07:00
Mike Danese 01590a9704 Merge pull request #27 from justinsb/upup_aws_image_aliases 
upup: Map matching image ids to source name
 2016-05-17 09:34:42 -07:00
Mike Danese 026bb16e49 Merge pull request #26 from justinsb/upup_aws_instance_userdata 
upup: Fetch instance userdata on AWS
 2016-05-17 09:34:30 -07:00
Mike Danese 7466c48a75 Merge pull request #25 from justinsb/upup_dryrun_print_interface 
upup: fix dryrun spurious printing of interface values
 2016-05-17 09:34:17 -07:00
Mike Danese d5928df6a4 Merge pull request #24 from justinsb/upup_aws_iamrole_policies 
upup: Perform JSON comparison on IAMRole PolicyDocuments
 2016-05-17 09:34:03 -07:00
Mike Danese cf98b75d39 Merge pull request #23 from justinsb/upup_aws_instance_tags 
upup: Discover tags on AWS instances
 2016-05-17 09:33:50 -07:00
Mike Danese 03ae922b01 Merge pull request #22 from justinsb/upup_aws_keypairs 
upup: Fix fingerprint calculation in AWS keypair
 2016-05-17 09:33:38 -07:00
Mike Danese a4344e1564 Merge pull request #19 from justinsb/upup_keypair 
upup: better keypair support
 2016-05-17 09:33:24 -07:00
Mike Danese d6026a431e Merge pull request #18 from justinsb/upup_reflection_redux 
upup: Make reflective walk more logical, use for dryrun change printing
 2016-05-17 09:33:16 -07:00
Mike Danese b48b503a7c Merge pull request #17 from justinsb/upup_gce_metadata 
upup: Discover metadata on GCE instance
 2016-05-17 09:33:11 -07:00
Mike Danese 5aa64c72a6 Merge pull request #16 from justinsb/upup_image_url 
upup: Map GCE image-url to string consistently
 2016-05-17 09:33:02 -07:00
Justin Santa Barbara c68aeec94d upup: tolerate aws eventual-consistency errors on tags 
AWS will sometimes return an error like "resource not found" when a
DescribeTags or CreateTags call immediately follows creation of the
resource.  Introduce a retry-loop when we get an error that is of the
appropriate type.
 2016-05-16 12:08:28 -04:00
Justin Santa Barbara 14ac983241 upup: Map matching image ids to source name 
This avoids spurious changes, and also is more intuitive for the user -
whatever name the user gave it, if it resolves to the same image, that
is the name we will use.
 2016-05-16 11:20:24 -04:00
Justin Santa Barbara a00e974a95 upup: Fetch instance userdata on AWS 
We simply weren't doing this until now
 2016-05-16 11:12:13 -04:00
Justin Santa Barbara e90f5c78e1 upup: fix dryrun spurious printing of interface values 
We missed the case where the interface changed (this only shows up if
other fields change also)
 2016-05-16 10:58:44 -04:00
Justin Santa Barbara ba77aea0ea upup: Perform JSON comparison on IAMRole PolicyDocuments 
AWS reformats them (inserting lots of whitespace), making a string
comparison incorrect.  Instead we parse to JSON and do a
reflect.DeepEqual check; if they are the same then we pretend the actual
value was the expected value.
 2016-05-15 23:49:01 -04:00
Justin Santa Barbara e6ce439fcd upup: Discover tags on AWS instances 
We simply weren't mapping them previously.
 2016-05-15 23:32:49 -04:00
Justin Santa Barbara 8752db39aa upup: Fix fingerprint calculation in AWS keypair 
Both fix the calculation itself to match AWS's weird fingerprint
algorithm, and also fix the comparison logic by which we infer that if
the fingerprint matches, that the public key matches also.
 2016-05-15 23:28:19 -04:00
Justin Santa Barbara 1c97a94d87 Rework keypair to fit our change model 
We also remove another special-case context (pki), so that it is just
another object type.
 2016-05-15 21:46:53 -04:00
Justin Santa Barbara c4742045ab Support changing GCE instance metadata 
This works, but is only a part-way there because:

1. We need to trigger cloudup
2. Our manifests don't pick up the change (e.g. kube-apiserver)
 2016-05-15 17:46:34 -04:00
Justin Santa Barbara ac220d2ba1 Allow Render methods to take fi.Context 
We call the Render methods on Tasks by reflection, and some of them
don't care about the Target, but do care about the Context (e.g. the PKI
tasks, which only care about the CAStore)
 2016-05-15 17:43:04 -04:00
Justin Santa Barbara cb7e20be3b Create HasAddress interface 
AWS ElasticIPs and GCE Addresses implement HasAddress.  This will be
used by dependent tasks to extract the IP address at runtime.
 2016-05-15 17:42:59 -04:00
Justin Santa Barbara 02d980c681 Make reflective walk more logical, use for dryrun change printing 
Remove a bunch of inconsistencies so that the reflective walk is not
suprising, and also rename it to ReflectRecursive.

Then use this for dry-run change printing.
 2016-05-15 17:41:19 -04:00
Justin Santa Barbara 0976ee1a73 Discover metadata on GCE instance 
We simply weren't mapping it before, which was causing spurious changes.
 2016-05-15 17:40:32 -04:00
Justin Santa Barbara 5a61c35052 Map GCE image-url to string consistently 
We need to reverse our image shortening consistently with how we resolve
the image, so that --dryrun does not report spurious changes.
 2016-05-15 17:39:26 -04:00
Justin Santa Barbara 370727aa44 Support for persistent disks, users & symlinks 
These are required to mount the master PD.  We use the kubernetes
safe-format-and-mount implementation.
 2016-05-13 13:29:56 -04:00
Justin Santa Barbara caccb8953f UpUp: AWS support 
Adds AWS support for both cloudup & nodeup.
Also cleaning up things found along the way!
 2016-05-09 13:08:27 -04:00
Justin Santa Barbara d4c2cfaae7 Initial version of upup: cloudup & nodeup 
* GCE support only
* Key and secret generation
* "Direct mode" makes API calls
* "Dry run mode" previews the changes
* Terraform output (though key generation not working for master ip)
* cloud-init output (though debian image does not ship with cloud-init)
 2016-05-06 16:01:33 -04:00