Commit Graph

111 Commits

Author SHA1 Message Date
Justin SB 4522a9bc66
Always log when a retry loop fails
We want to be sure the retry loop is working, and we want to know when
we're incurring retry failures (if something is expected to fail).
2018-12-21 14:16:51 -05:00
Justin SB 26bd75aecb
Bulk spelling fixes
Experimenting with my own spelling checker, these are the typos it caught.
2018-12-20 17:43:56 -05:00
Justin Santa Barbara 85d47cd67d s3: lazy-evaluate encryption policy
Should help performance a little bit, and should be a little faster.
2018-10-11 06:46:34 -07:00
Justin Santa Barbara 49e5797bc0 Google Cloud Storage md5 decoding fix
The MD5 is presented base64 encoded; we were trying to decode it as
hex.
2018-10-09 18:16:15 -07:00
k8s-ci-robot 66b9e0e8b0
Merge pull request #5726 from davidarcher/patch-1
Use appropriate log level for KOPS_STATE_S3_ACL debug message
2018-09-05 08:14:48 -07:00
k8s-ci-robot 2f1d2e07f7
Merge pull request #5565 from justinsb/refactor_printer
Refactor tables package to be more reusable
2018-09-03 15:28:36 -07:00
Levi Blackstone c4e2db4afc
Vendor servergroup module from gophercloud
* Bump gophercloud sha to f29afc2
* Add a prereq check for bazel and dep which is needed by `make dep-ensure`
* Document the process to add a vendored dependency
2018-08-30 11:25:54 -06:00
David Archer 83db56fab0
Use appropriate log level for KOPS_STATE_S3_ACL debug message 2018-08-30 09:58:23 -04:00
Justin Santa Barbara 1753423027 DigitalOcean: don't try to set SSE
We lost the p.sse check in a bad merge; restoring it here.

Fix #5519
2018-08-14 21:26:18 -04:00
Justin Santa Barbara 76f5ed2d9c Refactor tables package to be more reusable
We still need the reflect helpers, but we allow for clients to
register their own pretty-printers, which avoids the package
dependency for our pretty-printer.  We register our pretty printers in
an init function in the relevant package (in this case,
upup/pkg/fi/printers.go)

Fix #5551
2018-08-02 14:09:05 -04:00
Justin Santa Barbara 288c5aaf01 Add error handling (logging) when we fail to close a file
More missing error handling

Follows on from #5543
2018-07-28 16:50:13 -04:00
Mike Splain 9b691cdf3c Switch bucket encryption policy warning to debug 2018-06-22 14:53:33 -04:00
Rohith 4531384649 This PR attempts to solve two issues
a) The current implementation use's a static kubelet which doesn't not conform to the Node authorization mode (i.e. system:nodes:<nodename>)
b) As present the kubeconfig is static and reused across all the masters and nodes

The PR firstly introduces the ability for users to use bootstrap tokens and secondly when enabled ensure the kubelets for the masters as have unique usernames.  Note, this PR does not attempt to address the distribution of the bootstrap tokens themselves, that's for cluster admins. One solution for this would be a daemonset on the masters running on hostNetwork and reuse dns-controller to annotated the pods and give as the DNS

Notes:
- the master node do not use bootstrap tokens, instead given they have access to the ca anyhow, we generate certificates for each.
- when bootstrap token is not enabled the behaviour will stay the same; i.e. a kubelet configuration brought down from the store.
- when bootstrap tokens are enabled, the Nodes sit in a timeout loop waiting for the configuration to appear (by third party).
- given the nodeup docker and manifests builders are executed before the kubelet builder, the assumption here is a unit file kicks of a custom container to bootstrap the rest.
- the current firewalls on between the master and nodes are fairly open so no need to open ports between the two
- much of the work was ported from @justinsb PR [here](https://github.com/kubernetes/kops/pull/4134/)
- we add a very presumptuous server and client certificates for use with an authorizer (node-bootstrap-internal.dns_zone)

I do have an additional PR which performs the entire thing. The process being a node_authorizer which runs on the master nodes via a daemonset, the service implements a series of authorizers (i.e. alwaysallow, aws, gce etc). For aws, the process is similar to how vault authorizes nodes [here](https://www.vaultproject.io/docs/auth/aws.html). Nodeup no then calls out to the node_authorizer on bootstrap and provisions the kubelet.
2018-06-11 09:56:32 +01:00
k8s-ci-robot dd3381dc89
Merge pull request #5194 from chrisz100/feature/s3_bucket_encryption
Feature/s3 bucket encryption - Implements PR #4235
2018-06-10 15:32:01 -07:00
Justin Santa Barbara 4cea00ea75 Use HomeDir from client-go to get home directory
Works on windows & linux

Fixes #4523
2018-06-02 15:17:23 -04:00
Christian Jantz 6fba37ea63 Merge branch 'master' of github.com:gekart/kops into feature/s3_bucket_encryption 2018-05-23 10:49:21 +02:00
xh4n3 d25878f82f add String method for OSSFS to fix go vet issue 2018-04-04 15:24:33 +08:00
andrewsykim c82e3cf81a fix go vet error from util/pkg/vfs/ossfs.go 2018-04-03 18:00:19 -04:00
andrewsykim 54bee09f47 digitalocean: add kubelet hostname override 2018-04-03 01:16:50 -04:00
Xiao An 4aa68d2de9 a few updates based on suggestions
Signed-off-by: Xiao An <hac@zju.edu.cn>
2018-04-02 15:29:18 +08:00
xh4n3 49dd170eea include aliyun sdk 2018-04-02 15:24:22 +08:00
Xiao An 18e160748e add VFS implementation with Aliyun OSS
Signed-off-by: Xiao An <hac@zju.edu.cn>
2018-04-02 15:23:36 +08:00
andrewsykim 6fa37bf005 add digitalocean VFS 2018-04-01 23:05:46 -04:00
andrewsykim 2947bb1b9e allow s3 vfs scheme and sse to be configurable 2018-04-01 23:05:11 -04:00
Grischa Ekart 7c41e35bbc Implement AWS Default Bucket Encryption PR #4235 2018-03-07 23:26:28 +01:00
k8s-ci-robot 0ab8b57c2a
Merge pull request #4493 from justinsb/vfs_streaming
VFS: WriteFile takes an io.ReadSeeker
2018-02-26 15:50:45 -08:00
Mike Splain 45a57915e2 Fix bazel deprecation notice 2018-02-26 09:36:13 -05:00
Justin Santa Barbara 412cf377c2 VFS: WriteFile takes an io.ReadSeeker
Means we don't have to buffer big files in memory, in combination with
WriteTo for reading.
2018-02-26 09:09:17 -05:00
Mike Splain f40dc50a25 Update BUILD files to account for some recent changes 2018-02-12 17:16:33 -05:00
Kashif Saadat ac25853cd5 - Add etcdClusterSpec Image & Version in bootstrap data for Master nodes
- Reuse execWithTee fn for ETCD Command (tee & mkfifo in different path for newer image versions)
2018-02-10 12:14:36 +00:00
Justin Santa Barbara 8ef705353e Update gazelle 2018-02-03 13:27:23 -05:00
k8s-ci-robot cc67497776
Merge pull request #4246 from ottoyiu/s3_vfs
Improve S3 url parsing for vfsPath to support more naming conventions
2018-01-29 05:34:34 -08:00
Justin Santa Barbara 82b9a54332 VFS: Recognize file:// paths 2018-01-27 15:03:05 -08:00
Justin Santa Barbara d92bd77ccf VFS: Fix bug in CopyTree when dest does not exist
This particularly happened with a filesystem destination file
2018-01-25 10:08:08 -05:00
k8s-ci-robot 46a6d256d3
Merge pull request #4318 from justinsb/bazel_5
Add missing BUILD.bazel for util/pkg/slice
2018-01-23 22:23:30 -08:00
Justin Santa Barbara e2f91917d9 ReadTree: clarify that returns only files
Because the primary use-case is S3-style stores, we haven't really used
directories.  If we have a use-case, we can always pass a boolean
parameter or create an alternative function.
2018-01-23 23:42:00 -05:00
Justin Santa Barbara 97ed0e7cbf Add missing BUILD.bazel for util/pkg/slice 2018-01-22 00:26:18 -05:00
Kashif Saadat e315c350be Implement ability to update Load Balancer subnets 2018-01-17 11:57:29 +00:00
Otto Yiu e4427e9672 improve S3 url parsing for vfsPath to support more naming conventions 2018-01-12 16:07:18 -08:00
Justin Santa Barbara 6f6bafb65e VFS: Support io.WriterTo interface
Allows us to handle much bigger files (no need to buffer in-memory)
2018-01-08 22:34:27 -05:00
Justin Santa Barbara ec8db8b78c Initial implementation of bundle command
The bundle command will support enrollment of a machine via SSH.
2018-01-04 18:55:28 -05:00
k8s-ci-robot fcc904f468
Merge pull request #4170 from chrislovecnm/bazel-improvements
Improving bazel make targets, adding a target for kops cli, bumping go_rules version
2018-01-04 08:48:31 -08:00
Justin Santa Barbara 8c23031561 Add roundtrip tests for certs & private keys
Travis should cover all our supported go versions.
2017-12-29 21:18:29 -05:00
chrislovecnm 4dd3bb1dea Updating bazel BUILD files with new go_rules version 2017-12-29 15:03:14 -07:00
chrislovecnm 7057aaf1bb Enabling the file assets
File assets and the SHA files are uploaded to the new location. Files
when are users uses s3 are upload public read only. The copyfile task
uses only the existing SHA value.

This PR include major refactoring of the use of URLs.  Strings are no
longer categnated, but converted into a URL struct and path.Join is
utlilized.

A new values.go file is included so that we can start refactoring more
code out of the "fi" package.

A
2017-12-17 15:26:57 -07:00
chrislovecnm 609e268a1d gazelle updates with new bazel version 2017-11-05 17:41:53 -07:00
chrislovecnm b6b2c74fec updating bazel files 2017-11-04 10:08:50 -06:00
chrislovecnm 1e418c3e13 more goimport updates 2017-11-04 10:03:02 -06:00
zengchen1024 bbfd1e18a3 implement vfs with openstack swift 2017-11-02 17:08:16 +08:00
chrislovecnm 8d1ee1fa16 updating files for goimports 2017-11-01 12:51:43 -06:00