kubernetes
/
kops
mirror of https://github.com/kubernetes/kops.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
22,410 Commits 31 Branches 256 Tags 544 MiB
Commit Graph

395 Commits

Author SHA1 Message Date
Ole Markus With 4bf0fae33a Add function for getting memfs location 2021-07-01 08:27:48 +02:00
John Gardiner Myers eb076e3713 Render managed files with Terraform 2021-06-28 12:15:15 -07:00
John Gardiner Myers ead0941ae8 Make SpecOverride append to slices 2021-06-14 14:01:22 -07:00
John Gardiner Myers 76fc012f96 Allow unsetting fields from the command line 2021-06-14 08:52:32 -07:00
Kubernetes Prow Robot 3c4b6068b9
Merge pull request #11649 from h3poteto/fix-jwks-location 
Fix jwks object path in S3 for IRSA
 2021-06-01 08:26:27 -07:00
AkiraFukushima d52ec60c02
Fix issuer and jwks object path for IRSA 2021-06-01 23:35:21 +09:00
John Gardiner Myers e896a8a215 Fix detection of virtual-hosted-style S3 urls in us-east-1 2021-05-31 19:07:56 -07:00
Kubernetes Prow Robot 3a376e9048
Merge pull request #11387 from johngmyers/aws-config 
Enable reading shared config when possibly from CLI
 2021-05-23 15:15:38 -07:00
Ciprian Hacman a39d829f1f Set canonical location for downloads to artifacts.k8s.io 
And remove the legacy location for downloads.
 2021-05-14 00:41:56 +03:00
John Gardiner Myers 23de00da6e Enable reading shared config when possibly from CLI 2021-05-05 22:08:54 -07:00
Ciprian Hacman e0eab51c5c Recognize Ubuntu 21.04 2021-04-27 12:54:42 +03:00
Peter Rifel 7c900b7fae Generate and upload keys.json + discovery.json to public store 
Generate and upload keys.json + discovery.json to public store

Don't enable anonymous auth on publicjwks

Remove tests that won't work using FS VFS anymore
 2021-03-19 20:03:26 +01:00
Kubernetes Prow Robot 2083133cfc
Merge pull request #11047 from bmelbourne/update-misc-go-mods 
Update Go modules to latest versions
 2021-03-15 00:11:04 -07:00
Peter Rifel 21389c8276
Cleanup some nodeup & protokube logging 
Also log a channels error that we're seeing on flatcar to help with troubleshooting
 2021-03-14 21:49:35 -05:00
Barry Melbourne e30bf1cf35 Update Go modules to latest versions 2021-03-14 15:08:27 +00:00
Bharath Vedartham e5aa8177b6 Add protokube and channels urls 2021-03-06 00:31:18 +05:30
Kubernetes Prow Robot 016b0e5500
Merge pull request #10732 from zetaab/feature/uagent 
add user agent to openstack api requests
 2021-02-06 23:53:12 -08:00
Jesse Haka 034dad258c modify names 2021-02-05 09:57:54 +02:00
Jesse Haka 41d04d8d4b add user agent to openstack api requests 2021-02-04 23:04:06 +02:00
Bharath Vedartham 49f3ab0703 Throw error if path being set by kops set is not present in struct 2021-01-31 12:00:42 +05:30
Kubernetes Prow Robot 95e9cbf8ab
Merge pull request #10566 from rifelpet/fs-err 
Fix file not found error detection in fs://
 2021-01-12 22:42:35 -08:00
Peter Rifel 0df5f6c24d
Fix file not found error detection in fs:// 2021-01-12 20:57:33 -06:00
Justin Santa Barbara 78b139465c Refactor and centralize distribution logic 
Use of a struct makes it more sustainable, centralizing into the
distribution package makes it simpler to follow.
 2021-01-05 11:50:23 -05:00
Justin SB b17e44b709 Recognize ubuntu 20.10 
Teach nodeup about ubuntu 20.10, including the unusual
/etc/resolv.conf configuration.
 2021-01-05 10:53:40 -05:00
Kenji Kaneda a61caea8d2 Add Azure support 
This commit contains all changes required to support Azure
(https://github.com/kubernetes/kops/issues/3957).
 2020-12-21 08:27:54 -08:00
Justin SB 7d9ff3ba96 Refactor MirroredAsset into mirrors package 
This means we can use MirroredAsset for nodeup without circular
dependencies.  Also removes a duplicate constant that was declared
twice.
 2020-12-19 18:39:09 -05:00
Kenji Kaneda 40c944aa5c Fix a typo in an error message returned from buildAzureBlobPath 
invalid Azure Blob schem -> invalid Azure Blob scheme
 2020-11-23 08:16:39 -08:00
Kenji Kaneda 4555c0b2df Add support of Azure Blob storage to VFS 
The schema is "azureblob".

azureClient provides two ways to set up credential. One approach is to
use an account key stored in env var. This approach is used when
accessing Blob from kops CLI. The second approach is to retrieve
credentials from Instance Metadata Service. This works only when
azureClient is created on a VM that has sufficient privilege to access
a specified blob. This approach is used from nodeup, etcd-manager,
etc.
 2020-11-19 10:47:03 -08:00
Jesse Haka 67d69f16a9 allow reauth for openstack client 2020-10-30 08:57:49 +02:00
Ole Markus With f6ce70e5c3 Minor fixes to swiftfs.go 2020-10-08 20:32:29 +02:00
Ciprian Hacman 0c6f1c733c Use all kops mirrors to determine artifacts hashes 2020-09-18 09:44:37 +03:00
Kubernetes Prow Robot a5fc8895dc
Merge pull request #9857 from hakman/detect-aws-region 
Detect AWS region for S3 inside containers
 2020-09-09 23:17:44 -07:00
John Gardiner Myers 1e92c7740c Map ENOENT to ErrNotExist in FSPath 2020-09-05 21:46:57 -07:00
Ciprian Hacman 32e6da7576 Detect AWS region for S3 inside containers 2020-09-02 06:41:12 +03:00
Kubernetes Prow Robot fc66e0161e
Merge pull request #9836 from justinsb/openstack_reauth 
Always use OpenStack Swift reauthentication
 2020-08-31 00:40:20 -07:00
Justin SB c63ce4b5ab Implement setter by reflection 
This means we no longer have to individually hard-code the `kops set`
fields, however we use the "language" we're now demonstrated.

We add tests to ensure we have parity with our existing (hard-coded)
setter logic.
 2020-08-30 09:59:52 -04:00
Justin SB d4480e4721 Always use OpenStack Swift reauthentication 
If we were using credentials from env vars, we would not do
reauthentication with Swift.
 2020-08-29 08:25:59 -04:00
Ciprian Hacman 0da3980865 Use /etc/os-release to identify the distribution 2020-08-17 07:25:44 +03:00
Ciprian Hacman e29b84da01 Add tests for distributions 2020-08-17 07:25:43 +03:00
Ciprian Hacman 22ec1512dc Use numbers for distribution names 2020-08-17 07:25:43 +03:00
Ciprian Hacman 3825f657cc Use const for architectures and distributions 2020-08-17 07:25:43 +03:00
Ciprian Hacman 795373a499 Remove unused function 2020-08-17 07:25:43 +03:00
Ciprian Hacman 3f43d047dd Remove confusing comment 2020-08-17 07:25:43 +03:00
Ciprian Hacman e68ee80a93 Move and rename the "distros" package 2020-08-17 07:25:43 +03:00
Peter Rifel 4d9f0128a3
Upgrade to klog2 
This splits up the kubernetes 1.19 PR to make it easier to keep up to date until we get it sorted out.
 2020-08-16 20:56:48 -05:00
Ole Markus With a708a96c05 Adds support for using OS application credentials 
Application credentials allows you to export a purpose-specific set of
credentials for a user instead of exposing user login credentials.
Especially useful when using LDAP or similar for Openstack users.
Also lets you rotate credentials more easily since multiple application
credentials can be provisioned per user.

Update pkg/model/bootstrapscript.go

Co-authored-by: Ciprian Hacman <ciprianhacman@gmail.com>
 2020-08-07 14:26:47 +02:00
Ole Markus With d1479fb666 Add support for reading openstack metadata in vfs 2020-08-04 08:22:00 +02:00
John Gardiner Myers fed5587d77 Improve locking in memfs 2020-07-19 16:01:47 -07:00
Kubernetes Prow Robot aa7b67124f
Merge pull request #9535 from hakman/env-arch-amd64 
Force single arch support via env var
 2020-07-09 09:00:31 -07:00
Ciprian Hacman 3c84d83d37 Address review comments 2020-07-09 17:07:37 +03:00
Zhou Hao d6695b822f Add err judgment to os.RemoveAll 
Signed-off-by: Zhou Hao <zhouhao@cn.fujitsu.com>
 2020-07-09 16:48:35 +08:00
Zhou Hao 34931ed930 Add err judgment to ioutil.TempDir 
Signed-off-by: Zhou Hao <zhouhao@cn.fujitsu.com>
 2020-07-09 16:45:12 +08:00
Ciprian Hacman 9b77f372f1 Fix typo in function name 2020-07-09 10:45:50 +03:00
Ciprian Hacman 1f296e58f6 Force AMD64 only support via env var 2020-07-09 09:31:54 +03:00
Li Zhijian c3fc293ede cleanup tempfiles for fs_test 
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
 2020-07-02 10:09:34 +08:00
Xiaoyu Zhong 98c35cd220 Rename accessKeyId to accessKeyID 2020-06-22 09:32:20 +08:00
Kubernetes Prow Robot 8b371acef0
Merge pull request #9094 from olemarkus/vault-vfs 
Implement VFS for vault
 2020-06-20 12:02:39 -07:00
Ciprian Hacman a49879c4bc ARM64 support - Add architecture 2020-06-19 04:42:11 +03:00
Ole Markus With acaa1e1dfc Implement VFS for vault 2020-06-18 13:02:37 +02:00
ZouYu 2fc52ec6be fix some go-lint warning 
Signed-off-by: ZouYu <zouy.fnst@cn.fujitsu.com>
 2020-06-09 08:52:50 +08:00
Ciprian Hacman 654a0d2d8a Detect supported architecture during node setup 2020-06-03 17:23:59 +03:00
Kubernetes Prow Robot 5e27f74dd8
Merge pull request #9228 from justinsb/paginate_delete_all_versions 
S3 DeleteAllVersions: use pagination
 2020-05-31 15:59:54 -07:00
Justin SB 319ddcc333 S3 DeleteAllVersions: use pagination 
This way we're not limited to one page of versions.  This is likely a
purely theoretical concern, at least as we're using it today.
 2020-05-31 18:21:05 -04:00
Justin SB 7d7b8969ea Use AWS SDK to fetch metadata 
Previously the EC2 metadata service was straightforward HTTP, but IMDS
v2 now requires managing a session token (and is more secure for it).

We now use the AWS SDK when retrieving metadata; it automatically
supports IMDS v2.
 2020-05-31 17:23:01 -04:00
Ciprian Hacman d54aadc89c Fix nits for removal of S3 file versions 2020-05-28 06:50:32 +03:00
Kubernetes Prow Robot 92f8e22002
Merge pull request #9174 from johngmyers/remove-vfsscan 
Remove unused VFSScan
 2020-05-27 09:24:48 -07:00
Ciprian Hacman 9675692b84 Implement RemoveAll() for S3 paths 2020-05-25 07:46:32 +03:00
John Gardiner Myers 62ebbc5a5d Remove unused VFSScan 2020-05-24 21:27:28 -07:00
Ciprian Hacman b565122875 Remove delete markers also from S3 bucket 2020-05-24 17:42:02 +03:00
Ciprian Hacman a48ccfa06c Return warning instead of error to hide issues during cluster teardown 2020-05-24 15:20:20 +03:00
Ciprian Hacman 1a38a3feaa Return os.ErrNotExist when no versions are found 2020-05-24 11:42:18 +03:00
Ciprian Hacman 56af880c53 Remove TODO that was not addressed for a long time 2020-05-24 10:11:56 +03:00
Ciprian Hacman 831e3f0e57 Remove all versions of a file form the S3 bucket 2020-05-24 08:38:46 +03:00
Justin SB 5ed11fd9c7 GCE: don't rely on hostname being correct 
Distros that use systemd for DHCP often don't have the hostname
correct, due to e.g. the requirement for policy kit.

We don't rely on it being set correctly on other clouds; no real
reason to require it on GCP either!
 2020-05-17 15:20:58 -04:00
ZouYu ce8e61866f add unit test for util/pkg/hashing/hash.go 
Signed-off-by: ZouYu <zouy.fnst@cn.fujitsu.com>
 2020-05-11 14:10:02 +08:00
Kubernetes Prow Robot 5fc7ee69da
Merge pull request #8997 from littleroad/add_unit_test 
util/pkg/vfs/fs.go: Add Unit Test for WriteTo
 2020-05-04 17:29:08 -07:00
Johannes Würbach b92ef68bd6
Support S3 Virtual Hosted Style 2020-05-03 07:44:44 +02:00
Lu Fengqi f7990cad35 util/pkg/vfs/fs.go: Add Unit Test for WriteTo 
Signed-off-by: Lu Fengqi <lufq.fnst@cn.fujitsu.com>
 2020-04-27 17:46:07 +08:00
Justin Santa Barbara 108d1eee5d Replace deprecated x/net/context with context 
It's not x-perimental any more!
 2020-04-09 23:58:19 -04:00
Dao Cong Tien af6ff9b50d Add UT for util/pkg/vfs/memfs.go 
Signed-off-by: Dao Cong Tien <tiendc@vn.fujitsu.com>
 2020-03-12 19:04:24 +07:00
Kubernetes Prow Robot 5c01bff889
Merge pull request #8694 from johngmyers/fix-fileassets 
Fix uploading of file assets
 2020-03-10 08:23:43 -07:00
Kubernetes Prow Robot b65031f945
Merge pull request #8712 from truongnh1992/unittest 
Adding Unit Test for util/pkg/exec/exec.go
 2020-03-10 06:45:36 -07:00
Nguyen Hai Truong ed439cf57f Adding Unit Test for util/pkg/exec/exec.go 
Signed-off-by: Nguyen Hai Truong <truongnh@vn.fujitsu.com>
 2020-03-10 15:03:30 +07:00
Kubernetes Prow Robot 4308ce2af8
Merge pull request #8640 from tiendc/unit_test_util_pkg_proxy 
Add UT for util/pkg/proxy/proxy.go
 2020-03-09 23:27:36 -07:00
John Gardiner Myers fc7a955bfe Fix uploading of file assets 2020-03-09 20:00:36 -07:00
tiendc ce134f71b9
Update fs_test.go 2020-03-09 15:45:12 +07:00
tiendc 293233248c
Update fs_test.go 
Update fs_test.go
 2020-03-09 15:32:03 +07:00
Dao Cong Tien b95a24d43e Add UT for util/pkg/vfs/fs.go 
Signed-off-by: Dao Cong Tien <tiendc@vn.fujitsu.com>
 2020-03-06 13:43:24 +07:00
Dao Cong Tien e6e54d29e8 Add unit test for util/pkg/proxy/proxy.go 
Signed-off-by: Dao Cong Tien <tiendc@vn.fujitsu.com>
 2020-02-28 11:02:50 +07:00
Jesse Haka 11fa7b4b19 add s3 region 2020-02-20 14:50:54 +02:00
Kubernetes Prow Robot a34ad252ff
Merge pull request #8496 from justinsb/log_acls 
GCS: Log ACLs if we're writing them
 2020-02-06 22:49:43 -08:00
Justin SB 9fb80f9048
GCS: Log ACLs if we're writing them 
We log at V(4) because they are fairly verbose.
 2020-02-06 14:46:41 -05:00
Justin SB 9e7a026332
GCS: Fix bug where around retry on GCS 
We were recomputing the MD5, but we would need to rewind the stream first.
 2020-02-06 14:45:39 -05:00
Kubernetes Prow Robot b356bd4dc7
Merge pull request #6465 from ari-becker/bugfix/allow-local-filesystem-state-store 
Allow local filesystem state stores (to aid CI pull-request workflows)
 2020-01-17 10:52:26 -08:00
Justin Santa Barbara 5ebbfc96b9
Replace deprecated method calls to google cloud libraries 
Required for static-check to pass.
 2020-01-17 06:38:43 -05:00
Zhou Hao eff94028dd add unit test for Contains 
Signed-off-by: Zhou Hao <zhouhao@cn.fujitsu.com>
 2019-12-30 09:37:19 +08:00
Zhou Hao 0ea6d02c54 add unit test for GetUniqueStrings 
Signed-off-by: Zhou Hao <zhouhao@cn.fujitsu.com>
 2019-12-29 15:16:40 +08:00
Ari Becker 3236ba135b
Allow local filesystem state stores 2019-12-29 09:12:51 +02:00
Kubernetes Prow Robot be6e8a83e2
Merge pull request #8194 from bittopaz/ali-patch-2 
Alicloud: refine Alicloud RAM role policy
 2019-12-27 09:30:23 -08:00
Xiaoyu Zhong 5287f6d024 Refine Alicloud RAM role policy 2019-12-25 11:02:41 +08:00
tanjunchen 3f9400a588 util/pkg/vfs/:staticcheck 2019-12-23 10:20:56 +08:00
tanjunchen e4302e3630 util/pkg/: simplify code and remove unused code 2019-12-17 09:44:18 +08:00
Xiaoyu Zhong e580c5fff7 Alicloud: allow use RAM role for OSS client 2019-12-04 10:44:41 +08:00
feifei.zhang@huawei.com 48ebd260d3 fix golint failures 2019-11-24 16:38:58 +08:00
hwdef 3264e3b69d fix static check 2019-10-28 10:24:30 +08:00
Xiaoyu Zhong 002ddbb270 Alicloud: add hostname override 2019-10-16 21:53:40 +08:00
Kubernetes Prow Robot 21240d9da4
Merge pull request #7744 from tanjunchen/fix-up-code-bug 
fix-up some staticcheck error
 2019-10-06 11:45:08 -07:00
tanjunchen 8fe36dc72c fix-up some staticcheck error 2019-10-06 10:40:13 +08:00
tanjunchen 119e36be29 simplfy code 2019-10-04 22:19:08 +08:00
Kubernetes Prow Robot d6592fea47
Merge pull request #7560 from tanjunchen/fix-up-ineffectual-assignment 
ineffectual assignment to
 2019-10-01 06:33:34 -07:00
chentanjun 9e10230ccc fix-up-ineffectual-assignment 2019-09-29 17:22:07 +08:00
Peter Rifel 466ca95243 Dont assign unused values to variables 2019-09-27 12:51:19 -07:00
Justin SB cdaa7a3a48
Fix boilerplate: headers & packages 2019-09-25 12:48:14 -04:00
Justin SB 728e582360
Fill out kops controller functionality 
k8s 1.16 requires that we move label setting away from the kubelet, to
a central controller.  kops-controller is that controller.
 2019-09-25 12:04:34 -04:00
mikesplain 9e55b8230a Update copyright notices 
Also cleans some white spaces
 2019-09-09 14:47:51 -04:00
Kubernetes Prow Robot 9e8c0b4668
Merge pull request #7482 from beautytiger/static_check_fix 
fix static check error in vfssync.go
 2019-09-06 20:19:31 -07:00
Justin SB 3fbc906cbc
Create env-var helper function 
Refactor to start to centralize the env-var configuration for system
components, also start to add test coverage so we can be sure we
haven't broken things!
 2019-09-02 10:26:09 -07:00
Guangming Wang ad752f4887 fix static check error in vfssync.go 2019-09-01 15:33:31 +08:00
Lars Lehtonen 420273b309 util/pkg/vfs: Fix swallowed errors 2019-08-28 14:03:57 -07:00
Justin SB eca2ac6b80
Look for sha256 and sha1 files for artifacts 2019-08-23 18:26:25 -04:00
Kashif Saadat 2b61ace49c goimports update 2019-07-03 16:43:20 +01:00
Austin Moore 67d9f5f190
Move getProxyEnvVars into a util package 2019-06-05 15:59:19 -04:00
Justin SB 93f0b914cf
S3 VFS: Default to current region from metadata service 
We need a region to start from to make AWS calls.  us-east-1 works for
most credentials, but not for cn-north-1 credentials.  Instead, we get
the current region from metadata when running on EC2; and we continue
to fall-back to us-east-1.

For CLI commands (kops) the user will still have to set AWS_REGION,
but for system binaries (nodeup, etcd-manager), this should default
appropriately.

Note that the region doesn't have to be the actual region of the
bucket, just a region we can access.

Issue #6098
 2019-05-13 02:33:21 -07:00
Justin SB 76d03b3f71
Generated files: glog -> klog 2019-05-06 12:56:03 -04:00
Justin SB 3e33ac7682
Change code from glog to klog 
We don't call klog.InitFlags yet, because that will cause a flag
redefinition error until we get everyone to stop using glog.  That
will happen when we update to k8s 1.13.
 2019-05-06 12:54:51 -04:00
xichengliudui 3cd5c71330 Using const() defines constants together (part:3) 2019-04-11 15:19:27 -04:00
Gennady Trafimenkov acf9c9f016 Update bazel configuration 2019-03-03 16:11:33 +03:00
Gennady Trafimenkov 15bd566746 Correcly handle CRLF in the manifest 
kops replace/create/delete now correctly handle crlf in the manifest.

This fixes issue #6532
 2019-03-03 14:33:49 +03:00
Rohith b2bb67151b - fixing up the go formatting issues (a go v1.11 issue) 
- fixing up the unit test and import of require
 2019-02-04 11:59:43 +00:00
Rohith 39db0816df - updating the aws tasks to include a launch template resource 
- updating the autoscaling group tasks to provision mixed instance policies and templates
 2019-02-04 11:54:57 +00:00
Derek Lemon -T (delemon - AEROTEK INC at Cisco) 4e752ca62d Openstack Environment Variable Mapping 2019-01-15 14:21:41 -07:00
Derek Lemon -T (delemon - AEROTEK INC at Cisco) d0713c633a Use gophercloud to configure environment authentication 2019-01-15 14:21:31 -07:00
Derek Lemon -T (delemon - AEROTEK INC at Cisco) fb0939af9b Openstack Model, tasks, and cloud ops refactor 2019-01-15 14:16:08 -07:00
Jon Perritt 3064f6be15 server groups, lb, instance, and dns tasks, models and resources 2019-01-15 14:06:54 -07:00
Justin SB 4522a9bc66
Always log when a retry loop fails 
We want to be sure the retry loop is working, and we want to know when
we're incurring retry failures (if something is expected to fail).
 2018-12-21 14:16:51 -05:00
Justin SB 26bd75aecb
Bulk spelling fixes 
Experimenting with my own spelling checker, these are the typos it caught.
 2018-12-20 17:43:56 -05:00
Justin Santa Barbara 85d47cd67d s3: lazy-evaluate encryption policy 
Should help performance a little bit, and should be a little faster.
 2018-10-11 06:46:34 -07:00
Justin Santa Barbara 49e5797bc0 Google Cloud Storage md5 decoding fix 
The MD5 is presented base64 encoded; we were trying to decode it as
hex.
 2018-10-09 18:16:15 -07:00
k8s-ci-robot 66b9e0e8b0
Merge pull request #5726 from davidarcher/patch-1 
Use appropriate log level for KOPS_STATE_S3_ACL debug message
 2018-09-05 08:14:48 -07:00
k8s-ci-robot 2f1d2e07f7
Merge pull request #5565 from justinsb/refactor_printer 
Refactor tables package to be more reusable
 2018-09-03 15:28:36 -07:00
Levi Blackstone c4e2db4afc
Vendor servergroup module from gophercloud 
* Bump gophercloud sha to f29afc2
* Add a prereq check for bazel and dep which is needed by `make dep-ensure`
* Document the process to add a vendored dependency
 2018-08-30 11:25:54 -06:00
David Archer 83db56fab0
Use appropriate log level for KOPS_STATE_S3_ACL debug message 2018-08-30 09:58:23 -04:00
Justin Santa Barbara 1753423027 DigitalOcean: don't try to set SSE 
We lost the p.sse check in a bad merge; restoring it here.

Fix #5519
 2018-08-14 21:26:18 -04:00
Justin Santa Barbara 76f5ed2d9c Refactor tables package to be more reusable 
We still need the reflect helpers, but we allow for clients to
register their own pretty-printers, which avoids the package
dependency for our pretty-printer.  We register our pretty printers in
an init function in the relevant package (in this case,
upup/pkg/fi/printers.go)

Fix #5551
 2018-08-02 14:09:05 -04:00
Justin Santa Barbara 288c5aaf01 Add error handling (logging) when we fail to close a file 
More missing error handling

Follows on from #5543
 2018-07-28 16:50:13 -04:00
Mike Splain 9b691cdf3c Switch bucket encryption policy warning to debug 2018-06-22 14:53:33 -04:00
Rohith 4531384649 This PR attempts to solve two issues 
a) The current implementation use's a static kubelet which doesn't not conform to the Node authorization mode (i.e. system:nodes:<nodename>)
b) As present the kubeconfig is static and reused across all the masters and nodes

The PR firstly introduces the ability for users to use bootstrap tokens and secondly when enabled ensure the kubelets for the masters as have unique usernames.  Note, this PR does not attempt to address the distribution of the bootstrap tokens themselves, that's for cluster admins. One solution for this would be a daemonset on the masters running on hostNetwork and reuse dns-controller to annotated the pods and give as the DNS

Notes:
- the master node do not use bootstrap tokens, instead given they have access to the ca anyhow, we generate certificates for each.
- when bootstrap token is not enabled the behaviour will stay the same; i.e. a kubelet configuration brought down from the store.
- when bootstrap tokens are enabled, the Nodes sit in a timeout loop waiting for the configuration to appear (by third party).
- given the nodeup docker and manifests builders are executed before the kubelet builder, the assumption here is a unit file kicks of a custom container to bootstrap the rest.
- the current firewalls on between the master and nodes are fairly open so no need to open ports between the two
- much of the work was ported from @justinsb PR [here](https://github.com/kubernetes/kops/pull/4134/)
- we add a very presumptuous server and client certificates for use with an authorizer (node-bootstrap-internal.dns_zone)

I do have an additional PR which performs the entire thing. The process being a node_authorizer which runs on the master nodes via a daemonset, the service implements a series of authorizers (i.e. alwaysallow, aws, gce etc). For aws, the process is similar to how vault authorizes nodes [here](https://www.vaultproject.io/docs/auth/aws.html). Nodeup no then calls out to the node_authorizer on bootstrap and provisions the kubelet.
 2018-06-11 09:56:32 +01:00
k8s-ci-robot dd3381dc89
Merge pull request #5194 from chrisz100/feature/s3_bucket_encryption 
Feature/s3 bucket encryption - Implements PR #4235
 2018-06-10 15:32:01 -07:00
Justin Santa Barbara 4cea00ea75 Use HomeDir from client-go to get home directory 
Works on windows & linux

Fixes #4523
 2018-06-02 15:17:23 -04:00
Christian Jantz 6fba37ea63 Merge branch 'master' of github.com:gekart/kops into feature/s3_bucket_encryption 2018-05-23 10:49:21 +02:00
xh4n3 d25878f82f add String method for OSSFS to fix go vet issue 2018-04-04 15:24:33 +08:00
andrewsykim c82e3cf81a fix go vet error from util/pkg/vfs/ossfs.go 2018-04-03 18:00:19 -04:00
andrewsykim 54bee09f47 digitalocean: add kubelet hostname override 2018-04-03 01:16:50 -04:00
Xiao An 4aa68d2de9 a few updates based on suggestions 
Signed-off-by: Xiao An <hac@zju.edu.cn>
 2018-04-02 15:29:18 +08:00
xh4n3 49dd170eea include aliyun sdk 2018-04-02 15:24:22 +08:00
Xiao An 18e160748e add VFS implementation with Aliyun OSS 
Signed-off-by: Xiao An <hac@zju.edu.cn>
 2018-04-02 15:23:36 +08:00
andrewsykim 6fa37bf005 add digitalocean VFS 2018-04-01 23:05:46 -04:00
andrewsykim 2947bb1b9e allow s3 vfs scheme and sse to be configurable 2018-04-01 23:05:11 -04:00
Grischa Ekart 7c41e35bbc Implement AWS Default Bucket Encryption PR #4235 2018-03-07 23:26:28 +01:00
k8s-ci-robot 0ab8b57c2a
Merge pull request #4493 from justinsb/vfs_streaming 
VFS: WriteFile takes an io.ReadSeeker
 2018-02-26 15:50:45 -08:00
Mike Splain 45a57915e2 Fix bazel deprecation notice 2018-02-26 09:36:13 -05:00
Justin Santa Barbara 412cf377c2 VFS: WriteFile takes an io.ReadSeeker 
Means we don't have to buffer big files in memory, in combination with
WriteTo for reading.
 2018-02-26 09:09:17 -05:00
Mike Splain f40dc50a25 Update BUILD files to account for some recent changes 2018-02-12 17:16:33 -05:00
Kashif Saadat ac25853cd5 - Add etcdClusterSpec Image & Version in bootstrap data for Master nodes 
- Reuse execWithTee fn for ETCD Command (tee & mkfifo in different path for newer image versions)
 2018-02-10 12:14:36 +00:00
Justin Santa Barbara 8ef705353e Update gazelle 2018-02-03 13:27:23 -05:00
k8s-ci-robot cc67497776
Merge pull request #4246 from ottoyiu/s3_vfs 
Improve S3 url parsing for vfsPath to support more naming conventions
 2018-01-29 05:34:34 -08:00
Justin Santa Barbara 82b9a54332 VFS: Recognize file:// paths 2018-01-27 15:03:05 -08:00
Justin Santa Barbara d92bd77ccf VFS: Fix bug in CopyTree when dest does not exist 
This particularly happened with a filesystem destination file
 2018-01-25 10:08:08 -05:00
k8s-ci-robot 46a6d256d3
Merge pull request #4318 from justinsb/bazel_5 
Add missing BUILD.bazel for util/pkg/slice
 2018-01-23 22:23:30 -08:00
Justin Santa Barbara e2f91917d9 ReadTree: clarify that returns only files 
Because the primary use-case is S3-style stores, we haven't really used
directories.  If we have a use-case, we can always pass a boolean
parameter or create an alternative function.
 2018-01-23 23:42:00 -05:00
Justin Santa Barbara 97ed0e7cbf Add missing BUILD.bazel for util/pkg/slice 2018-01-22 00:26:18 -05:00
Kashif Saadat e315c350be Implement ability to update Load Balancer subnets 2018-01-17 11:57:29 +00:00
Otto Yiu e4427e9672 improve S3 url parsing for vfsPath to support more naming conventions 2018-01-12 16:07:18 -08:00
Justin Santa Barbara 6f6bafb65e VFS: Support io.WriterTo interface 
Allows us to handle much bigger files (no need to buffer in-memory)
 2018-01-08 22:34:27 -05:00
Justin Santa Barbara ec8db8b78c Initial implementation of bundle command 
The bundle command will support enrollment of a machine via SSH.
 2018-01-04 18:55:28 -05:00
k8s-ci-robot fcc904f468
Merge pull request #4170 from chrislovecnm/bazel-improvements 
Improving bazel make targets, adding a target for kops cli, bumping go_rules version
 2018-01-04 08:48:31 -08:00
Justin Santa Barbara 8c23031561 Add roundtrip tests for certs & private keys 
Travis should cover all our supported go versions.
 2017-12-29 21:18:29 -05:00
chrislovecnm 4dd3bb1dea Updating bazel BUILD files with new go_rules version 2017-12-29 15:03:14 -07:00
chrislovecnm 7057aaf1bb Enabling the file assets 
File assets and the SHA files are uploaded to the new location. Files
when are users uses s3 are upload public read only. The copyfile task
uses only the existing SHA value.

This PR include major refactoring of the use of URLs.  Strings are no
longer categnated, but converted into a URL struct and path.Join is
utlilized.

A new values.go file is included so that we can start refactoring more
code out of the "fi" package.

A
 2017-12-17 15:26:57 -07:00
chrislovecnm 609e268a1d gazelle updates with new bazel version 2017-11-05 17:41:53 -07:00
chrislovecnm b6b2c74fec updating bazel files 2017-11-04 10:08:50 -06:00
chrislovecnm 1e418c3e13 more goimport updates 2017-11-04 10:03:02 -06:00
zengchen1024 bbfd1e18a3 implement vfs with openstack swift 2017-11-02 17:08:16 +08:00
chrislovecnm 8d1ee1fa16 updating files for goimports 2017-11-01 12:51:43 -06:00
Justin Santa Barbara 2de6538692 Clarify comment on ReadTree 
Make it clearer that it needs to fetch all the files recursively
 2017-10-30 23:48:40 -04:00
Justin Santa Barbara b2bcba4a6d GCE: Use object-level permissions for files in GCS 
This lets us configure cross-project permissions while ourselves needing
minimal permissions, but also gives us a nice hook for future lockdown
of object-level permissions.
 2017-10-29 19:17:00 -04:00
Justin Santa Barbara d1ee8026ac GCE: Tasks for object & bucket level permissions 
We also switch to setting a bucket-level ACL permission, as this
requires less permissions.
 2017-10-29 18:08:08 -04:00
Justin Santa Barbara dbbe3f373b GCE: Set up permissions for cross-project configurations 
This ensures that the cluster can read the kops state store files, even
if the GCS bucket is in a different project.

We automatically set up an IAM access policy that grants access.
 2017-10-28 03:24:18 -04:00
Kubernetes Submit Queue 4d345d18d8 Merge pull request #3461 from justinsb/gcs_retry_on_error 
Automatic merge from submit-queue.

GCS paths; retry on error
 2017-10-08 14:18:59 -07:00
Justin Santa Barbara 0143be7c4f autogen: BUILD and BUILD.bazel 2017-10-02 14:27:21 -04:00
Justin Santa Barbara 83300fc39f GCS paths; retry on error 
The AWS SDK does this for us, I think, the GS SDK does not.
 2017-09-27 09:31:33 -04:00
Justin Santa Barbara 559d885480 Mirror keystore & secretstore 
This allows us to have our API objects in kops-server, but our
configuration on S3 or GCS.
 2017-09-24 00:09:02 -04:00
Justin Santa Barbara d257e73b1c GCS: Don't reuse same error message 
We had exactly the same error message for two code paths, which made
figuring out the cause harder.
 2017-09-16 21:39:14 -04:00
Kubernetes Submit Queue 593f444297 Merge pull request #3359 from justinsb/delete_more_secrets 
Automatic merge from submit-queue

Support for deleting tokens & keypairs
 2017-09-14 19:30:08 -07:00
Justin Santa Barbara 106875115d Support for deleting tokens & keypairs 
This now allows for deleting all secrets, which means we can have a
procedure for rotating all keys.
 2017-09-09 01:04:45 -04:00
Rohith aaf6143a98 Toolbox Templating 
Extending the current implementation of toolbox template to include multiple files and snippets. Note, i've removed the requirements for defaults as I think people should be forced to specifically pass them.

- fixing the vetting iseues to the method YamlToJson -> YAMLToJSON
- adding a safety check to ensure templates don't reference an unknown value
- extending the unit test to ensure the above works on main and snippets
- include the ability to specify multiple configuration files, useful for common.yaml and prod.yaml etc

Requested Changes - Toolbox Templating

Added the requested changes

- moved the templater into it's own package rather than using base util
- moved to using the sprig library for additional template function
- @note: i couldn't find a native way in sprig to do snippets, also the i've overloaded the indent as it appears to do the indent on all lines rather than on the newline, meaning i'd have to shift my first line back by the indent to get it to work, which seems ugly
 2017-09-08 20:30:21 +01:00
Justin Santa Barbara 1ac08b5cf1 Add missed error handling on session.NewSession 2017-08-28 07:52:11 -04:00
Justin Santa Barbara a26b28576e AWS: Always use verbose errors 
As otherwise very difficult to diagnose errors
 2017-08-25 23:08:39 -04:00
Lars Lehtonen a9bbe3af24
Wrap AWS NewSession() errors in vfs package 2017-08-25 13:25:04 -07:00
Lars Lehtonen afea9d05c5
Replace deprecated aws session.New() with session.NewSession() 2017-08-22 17:28:55 -07:00