kubernetes
/
kops
mirror of https://github.com/kubernetes/kops.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,992 Commits 31 Branches 256 Tags 544 MiB
Commit Graph

151 Commits

Author SHA1 Message Date
Guangming Wang ad752f4887 fix static check error in vfssync.go 2019-09-01 15:33:31 +08:00
Lars Lehtonen 420273b309 util/pkg/vfs: Fix swallowed errors 2019-08-28 14:03:57 -07:00
Justin SB eca2ac6b80
Look for sha256 and sha1 files for artifacts 2019-08-23 18:26:25 -04:00
Justin SB 93f0b914cf
S3 VFS: Default to current region from metadata service 
We need a region to start from to make AWS calls.  us-east-1 works for
most credentials, but not for cn-north-1 credentials.  Instead, we get
the current region from metadata when running on EC2; and we continue
to fall-back to us-east-1.

For CLI commands (kops) the user will still have to set AWS_REGION,
but for system binaries (nodeup, etcd-manager), this should default
appropriately.

Note that the region doesn't have to be the actual region of the
bucket, just a region we can access.

Issue #6098
 2019-05-13 02:33:21 -07:00
Justin SB 76d03b3f71
Generated files: glog -> klog 2019-05-06 12:56:03 -04:00
Justin SB 3e33ac7682
Change code from glog to klog 
We don't call klog.InitFlags yet, because that will cause a flag
redefinition error until we get everyone to stop using glog.  That
will happen when we update to k8s 1.13.
 2019-05-06 12:54:51 -04:00
xichengliudui 3cd5c71330 Using const() defines constants together (part:3) 2019-04-11 15:19:27 -04:00
Derek Lemon -T (delemon - AEROTEK INC at Cisco) 4e752ca62d Openstack Environment Variable Mapping 2019-01-15 14:21:41 -07:00
Derek Lemon -T (delemon - AEROTEK INC at Cisco) d0713c633a Use gophercloud to configure environment authentication 2019-01-15 14:21:31 -07:00
Derek Lemon -T (delemon - AEROTEK INC at Cisco) fb0939af9b Openstack Model, tasks, and cloud ops refactor 2019-01-15 14:16:08 -07:00
Jon Perritt 3064f6be15 server groups, lb, instance, and dns tasks, models and resources 2019-01-15 14:06:54 -07:00
Justin SB 4522a9bc66
Always log when a retry loop fails 
We want to be sure the retry loop is working, and we want to know when
we're incurring retry failures (if something is expected to fail).
 2018-12-21 14:16:51 -05:00
Justin SB 26bd75aecb
Bulk spelling fixes 
Experimenting with my own spelling checker, these are the typos it caught.
 2018-12-20 17:43:56 -05:00
Justin Santa Barbara 85d47cd67d s3: lazy-evaluate encryption policy 
Should help performance a little bit, and should be a little faster.
 2018-10-11 06:46:34 -07:00
Justin Santa Barbara 49e5797bc0 Google Cloud Storage md5 decoding fix 
The MD5 is presented base64 encoded; we were trying to decode it as
hex.
 2018-10-09 18:16:15 -07:00
k8s-ci-robot 66b9e0e8b0
Merge pull request #5726 from davidarcher/patch-1 
Use appropriate log level for KOPS_STATE_S3_ACL debug message
 2018-09-05 08:14:48 -07:00
Levi Blackstone c4e2db4afc
Vendor servergroup module from gophercloud 
* Bump gophercloud sha to f29afc2
* Add a prereq check for bazel and dep which is needed by `make dep-ensure`
* Document the process to add a vendored dependency
 2018-08-30 11:25:54 -06:00
David Archer 83db56fab0
Use appropriate log level for KOPS_STATE_S3_ACL debug message 2018-08-30 09:58:23 -04:00
Justin Santa Barbara 1753423027 DigitalOcean: don't try to set SSE 
We lost the p.sse check in a bad merge; restoring it here.

Fix #5519
 2018-08-14 21:26:18 -04:00
Justin Santa Barbara 288c5aaf01 Add error handling (logging) when we fail to close a file 
More missing error handling

Follows on from #5543
 2018-07-28 16:50:13 -04:00
Mike Splain 9b691cdf3c Switch bucket encryption policy warning to debug 2018-06-22 14:53:33 -04:00
k8s-ci-robot dd3381dc89
Merge pull request #5194 from chrisz100/feature/s3_bucket_encryption 
Feature/s3 bucket encryption - Implements PR #4235
 2018-06-10 15:32:01 -07:00
Justin Santa Barbara 4cea00ea75 Use HomeDir from client-go to get home directory 
Works on windows & linux

Fixes #4523
 2018-06-02 15:17:23 -04:00
Christian Jantz 6fba37ea63 Merge branch 'master' of github.com:gekart/kops into feature/s3_bucket_encryption 2018-05-23 10:49:21 +02:00
xh4n3 d25878f82f add String method for OSSFS to fix go vet issue 2018-04-04 15:24:33 +08:00
andrewsykim c82e3cf81a fix go vet error from util/pkg/vfs/ossfs.go 2018-04-03 18:00:19 -04:00
andrewsykim 54bee09f47 digitalocean: add kubelet hostname override 2018-04-03 01:16:50 -04:00
Xiao An 4aa68d2de9 a few updates based on suggestions 
Signed-off-by: Xiao An <hac@zju.edu.cn>
 2018-04-02 15:29:18 +08:00
xh4n3 49dd170eea include aliyun sdk 2018-04-02 15:24:22 +08:00
Xiao An 18e160748e add VFS implementation with Aliyun OSS 
Signed-off-by: Xiao An <hac@zju.edu.cn>
 2018-04-02 15:23:36 +08:00
andrewsykim 6fa37bf005 add digitalocean VFS 2018-04-01 23:05:46 -04:00
andrewsykim 2947bb1b9e allow s3 vfs scheme and sse to be configurable 2018-04-01 23:05:11 -04:00
Grischa Ekart 7c41e35bbc Implement AWS Default Bucket Encryption PR #4235 2018-03-07 23:26:28 +01:00
k8s-ci-robot 0ab8b57c2a
Merge pull request #4493 from justinsb/vfs_streaming 
VFS: WriteFile takes an io.ReadSeeker
 2018-02-26 15:50:45 -08:00
Mike Splain 45a57915e2 Fix bazel deprecation notice 2018-02-26 09:36:13 -05:00
Justin Santa Barbara 412cf377c2 VFS: WriteFile takes an io.ReadSeeker 
Means we don't have to buffer big files in memory, in combination with
WriteTo for reading.
 2018-02-26 09:09:17 -05:00
Justin Santa Barbara 8ef705353e Update gazelle 2018-02-03 13:27:23 -05:00
k8s-ci-robot cc67497776
Merge pull request #4246 from ottoyiu/s3_vfs 
Improve S3 url parsing for vfsPath to support more naming conventions
 2018-01-29 05:34:34 -08:00
Justin Santa Barbara 82b9a54332 VFS: Recognize file:// paths 2018-01-27 15:03:05 -08:00
Justin Santa Barbara d92bd77ccf VFS: Fix bug in CopyTree when dest does not exist 
This particularly happened with a filesystem destination file
 2018-01-25 10:08:08 -05:00
Justin Santa Barbara e2f91917d9 ReadTree: clarify that returns only files 
Because the primary use-case is S3-style stores, we haven't really used
directories.  If we have a use-case, we can always pass a boolean
parameter or create an alternative function.
 2018-01-23 23:42:00 -05:00
Otto Yiu e4427e9672 improve S3 url parsing for vfsPath to support more naming conventions 2018-01-12 16:07:18 -08:00
Justin Santa Barbara 6f6bafb65e VFS: Support io.WriterTo interface 
Allows us to handle much bigger files (no need to buffer in-memory)
 2018-01-08 22:34:27 -05:00
Justin Santa Barbara ec8db8b78c Initial implementation of bundle command 
The bundle command will support enrollment of a machine via SSH.
 2018-01-04 18:55:28 -05:00
k8s-ci-robot fcc904f468
Merge pull request #4170 from chrislovecnm/bazel-improvements 
Improving bazel make targets, adding a target for kops cli, bumping go_rules version
 2018-01-04 08:48:31 -08:00
Justin Santa Barbara 8c23031561 Add roundtrip tests for certs & private keys 
Travis should cover all our supported go versions.
 2017-12-29 21:18:29 -05:00
chrislovecnm 4dd3bb1dea Updating bazel BUILD files with new go_rules version 2017-12-29 15:03:14 -07:00
chrislovecnm 7057aaf1bb Enabling the file assets 
File assets and the SHA files are uploaded to the new location. Files
when are users uses s3 are upload public read only. The copyfile task
uses only the existing SHA value.

This PR include major refactoring of the use of URLs.  Strings are no
longer categnated, but converted into a URL struct and path.Join is
utlilized.

A new values.go file is included so that we can start refactoring more
code out of the "fi" package.

A
 2017-12-17 15:26:57 -07:00
chrislovecnm 609e268a1d gazelle updates with new bazel version 2017-11-05 17:41:53 -07:00
chrislovecnm b6b2c74fec updating bazel files 2017-11-04 10:08:50 -06:00
zengchen1024 bbfd1e18a3 implement vfs with openstack swift 2017-11-02 17:08:16 +08:00
chrislovecnm 8d1ee1fa16 updating files for goimports 2017-11-01 12:51:43 -06:00
Justin Santa Barbara 2de6538692 Clarify comment on ReadTree 
Make it clearer that it needs to fetch all the files recursively
 2017-10-30 23:48:40 -04:00
Justin Santa Barbara b2bcba4a6d GCE: Use object-level permissions for files in GCS 
This lets us configure cross-project permissions while ourselves needing
minimal permissions, but also gives us a nice hook for future lockdown
of object-level permissions.
 2017-10-29 19:17:00 -04:00
Justin Santa Barbara d1ee8026ac GCE: Tasks for object & bucket level permissions 
We also switch to setting a bucket-level ACL permission, as this
requires less permissions.
 2017-10-29 18:08:08 -04:00
Justin Santa Barbara dbbe3f373b GCE: Set up permissions for cross-project configurations 
This ensures that the cluster can read the kops state store files, even
if the GCS bucket is in a different project.

We automatically set up an IAM access policy that grants access.
 2017-10-28 03:24:18 -04:00
Kubernetes Submit Queue 4d345d18d8 Merge pull request #3461 from justinsb/gcs_retry_on_error 
Automatic merge from submit-queue.

GCS paths; retry on error
 2017-10-08 14:18:59 -07:00
Justin Santa Barbara 0143be7c4f autogen: BUILD and BUILD.bazel 2017-10-02 14:27:21 -04:00
Justin Santa Barbara 83300fc39f GCS paths; retry on error 
The AWS SDK does this for us, I think, the GS SDK does not.
 2017-09-27 09:31:33 -04:00
Justin Santa Barbara 559d885480 Mirror keystore & secretstore 
This allows us to have our API objects in kops-server, but our
configuration on S3 or GCS.
 2017-09-24 00:09:02 -04:00
Justin Santa Barbara d257e73b1c GCS: Don't reuse same error message 
We had exactly the same error message for two code paths, which made
figuring out the cause harder.
 2017-09-16 21:39:14 -04:00
Justin Santa Barbara 106875115d Support for deleting tokens & keypairs 
This now allows for deleting all secrets, which means we can have a
procedure for rotating all keys.
 2017-09-09 01:04:45 -04:00
Justin Santa Barbara 1ac08b5cf1 Add missed error handling on session.NewSession 2017-08-28 07:52:11 -04:00
Justin Santa Barbara a26b28576e AWS: Always use verbose errors 
As otherwise very difficult to diagnose errors
 2017-08-25 23:08:39 -04:00
Lars Lehtonen a9bbe3af24
Wrap AWS NewSession() errors in vfs package 2017-08-25 13:25:04 -07:00
Lars Lehtonen afea9d05c5
Replace deprecated aws session.New() with session.NewSession() 2017-08-22 17:28:55 -07:00
Lars Lehtonen 4e847ce020
Fix swallowed err variable in vfs package 2017-07-18 18:09:39 -07:00
Justin Santa Barbara e945322cab Support generated clientset as alternative to vfs clientset 
We modelled our VFS clientset (for API objects backed by a VFS path)
after the "real" clientsets, so now it is relatively easy to add a
second implementation that will be backed by a real clientset.

The snafu here is that we weren't really using namespaces previously.
Namespaces do seem to be the primary RBAC scoping mechanism though, so
we start using them with the real clientset.

The namespace is currently inferred from the cluster name.  We map dots
to dashes, because of namespace limitations, which could yield
collisions, but we'll deal with this by simply preventing users from
creating conflicting cluster names - i.e. you simply won't be able to
create a.b.example.com and a-b.example.com
 2017-06-20 22:29:37 -04:00
Miao Luo 22c49e76cb Fix user-defined s3 endpoint support. 
Address review feedbacks and remove unintended space.
 2017-04-19 23:46:05 -07:00
Miao Luo 76437a77d4 Support user-defined s3 endpoint. 2017-04-18 11:27:07 -07:00
yissachar 5b03f36b27 Merge pull request #2125 from justinsb/issue_2108 
More logging around errors in s3 write path
 2017-03-20 01:15:27 -04:00
yissachar e873950633 Merge pull request #2113 from justinsb/vfs_path_validation_error 
Fix error for invalid vfs paths
 2017-03-18 21:36:26 -04:00
Justin Santa Barbara cb4641fea3 Code updates 2017-03-16 02:40:50 -04:00
Justin Santa Barbara 8104ba2cea More logging around errors in s3 write path 
Hopefully to shed some light on issues like #2108
 2017-03-15 09:52:32 -04:00
Justin Santa Barbara 32b9e1b5b1 Fix error for invalid vfs paths 
We should be printing the path when it fails validation
 2017-03-13 10:42:31 -04:00
Justin Santa Barbara 3d14d07616 Support cloud-config on GCE 2017-02-28 20:08:03 -05:00
Justin Santa Barbara 645f330dad Re-enable GCE support 
We move everything to the models.  We feature-flag it, because we
probably want to change the names etc, and we aren't going to be able to
offer smooth upgrades until that is done.
 2017-02-28 20:08:03 -05:00
Seth Pollack b1702d749e
validate region 2017-02-23 12:03:03 -05:00
Justin Santa Barbara 1e5dab5a6c Actually retry on HTTP 404s 
We were returning the "don't retry" value (true).

Also tidy up the logic a little.
 2017-02-05 13:17:09 -05:00
Justin Santa Barbara 34cd84f5a7 Merge pull request #1705 from tazjin/fix/s3-location-call-timeout 
fix s3context: Attempt a normal S3 call before bruteforcing location
 2017-01-30 20:44:03 -05:00
Vincent Ambo 8c85935f1f
fix s3context: Attempt a normal S3 call before bruteforcing location 
In cases where the user is the bucket owner an initial call to
s3.GetBucketLocation will succeed. If it does return an error we
fall back to the bruteforce method.

This effectively makes the behaviour unchanged from previous versions
for bucket owners.
 2017-01-30 08:01:39 +01:00
Justin Santa Barbara 45a62a02aa s3 path: apply suffix to ReadTree also 
Otherwise we were matching directories with the same prefix.
 2017-01-29 21:35:26 -05:00
Kris Nova ca1ee3e1a2 Merge pull request #1247 from tazjin/fix/s3-cross-account 
Fix issues related to cross-account S3 bucket sharing
 2017-01-27 21:25:33 -07:00
Vincent Ambo cd6f9eb66a refactor s3context: Use Go time constants 2017-01-27 13:39:15 +01:00
Vincent Ambo 0728b7c9fa refactor vfs: Create location request further down 
Minor refactor, the request was created one level up originally
because I had added two separate steps for initially determining
whether we have to use the bruteforce method.

However this is a premature optimisation and unnecessary due to the
concurrency behaviour we've got now.
 2017-01-26 19:52:23 +01:00
Vincent Ambo 3b6e3bda56 chore: Run gofmt on new VFS code 2017-01-26 12:30:04 +01:00
Vincent Ambo 5a25a96c93 feat vfs: Fetch S3 bucket locations cross-account 
The AWS API makes it difficult to retrieve S3 bucket locations from shared buckets
with bucket-policy based access delegations. This introduces a workaround for the
issue.

AWS is aware of the issue but for the time being they can not provide information
about when it will be fixed.

See #1247 for more information.
 2017-01-26 11:47:09 +01:00
Vincent Ambo db0155b5b5 feat s3fs: Allow bucket object ACL override 
When sharing S3 buckets across accounts it may be necessary to override ACLs
per object to avoid locking out different accounts.

This commit lets users specify a `KOPS_STATE_S3_ACL` environment variable which
(if specified) overrides the ACL in the PutObject request.

Fixes #907
 2017-01-26 11:47:09 +01:00
Justin Santa Barbara bb42ae6723 Recognize file:// urls 2017-01-25 23:33:53 -05:00
Justin Santa Barbara 8c84ed3fe8 Retry readHttpLocation on 500 errors 
Fix #1441
 2017-01-15 20:59:02 -05:00
Justin Santa Barbara 0a56d3d2e1 Raise an error on an invalid s3 path 
Fix #902
 2017-01-08 14:37:46 -05:00
Justin Santa Barbara b36b75ad0c Always set CredentialsChainVerboseErrors when initializing AWS 
Fix #605
 2017-01-03 20:03:03 -05:00
Justin Santa Barbara afd0c25abe First model -> tf test 2016-12-11 17:11:10 -05:00
yancl aec534a93f change KOPS_AWS_REGION to AWS_REGION 2016-12-05 15:04:03 +00:00
yancl bc444fd535 fix the aws region that hardcoded to "us-east-1" which doesn't work in 
some isolated regions(cn-north-1 and GovCloud, for example)
 2016-11-21 08:53:45 +00:00
chrislovecnm 8fa2aac99f fixing more headers 2016-10-15 19:20:56 -06:00
Justin Santa Barbara defa53bb89 Fix S3 initialization 2016-10-07 01:52:01 -04:00
Justin Santa Barbara 767c9a6416 Fix tests; don't hit s3 until we need to 2016-10-07 01:38:44 -04:00
Justin Santa Barbara f7fa324858 Initial k8s-style kops API 
We try to emulate the k8s Clientset approach
 2016-10-07 01:13:02 -04:00
Justin Santa Barbara 39c3c85262 Skip directory objects in S3 when listing files 
Issue #520
Issue #548
 2016-09-30 10:07:15 -04:00