API servers also have access to secret store, so there is no need to go through kops-controller.
This lets API server only depend on etcd from the CP nodes, which should make it easier to scale out API servers under pressure
Without secure node auth enabled, commands like `kubectl logs` may fail
with certain configurations.
Previously, we checked if anonymousAuth was enabled on the kubelet
before securing node communication, but this isn't really relevant. We
can still authenticate even if anonymous access is allowed.
Ensure apiserver role can only be used on AWS (because of firewalling)
Apply api-server label to CP as well
Consolidate node not ready validation message
Guard apiserver nodes with a feature flag
Rename Apiserver role to APIServer
Add an integration test for apiserver nodes
Rename Apiserver role to APIServer
Enumerate all roles in rolling update docs
Apply suggestions from code review
Co-authored-by: Steven E. Harris <seh@panix.com>
As with the Cluster-level "spec.updatePolicy" field, add a similar
field at the InstanceGroup level, allowing overriding of the
cluster-level choice in each InstanceGroup.
Introduce a new value for the field ("automatic") as equivalent to the
default value applied when the field is absent. Honoring this new
value allows disabling automatic updates at the cluster level, but
then enabling them again for particular InstanceGroups. Without such a
positive affirmation, it's not possible to override a cluster-level
"external" policy at the InstanceGroup level, as there's no way to
specify positively that you want to recover the default
value. Instead, expressing the explicit "automatic" value is clear and
unambiguous.
Even if we don't install containerd (e.g. ContainerOS or Flatcar), we
likely still need to configure it; particularly in the case of
kubenet.
Additionally, on ContainerOS we can't change the path from
/etc/containerd/config.toml, so we have to write it there. We may in
future be able to use this on all distros.
Configure kubenet in containerd/CNI mode to match upstream configuration.
Biggest change is a move to the ptp plugin.
Co-authored-by: Ciprian Hacman <ciprian@hakman.dev>
Currently, kOps uses cgroupfs cgroup driver for the kubelet and CRIs. This PR defaults
the cgroup driver to systemd for clusters created with k8s versions >= 1.20.
Using systemd as the cgroup-driver is the recommended way as per
https://kubernetes.io/docs/setup/production-environment/container-runtimes/
Upstream bind mounts /var/lib/kubelet with exec, dev and suid
permissions, because emptyDirs end up inheriting these permissions.
Similarly, /home/kubernetes/flexvolume needs exec permission to
support flexdrivers.
kops-controller can now serve the instance group & cluster config to
nodes, as part of the bootstrap process.
This enables nodes to boot without access to the state
store (i.e. without S3 / GCS / etc permissions)
Feature-flagged behind the KopsControllerStateStore feature-flag.
We tend to build cloud, call some method, and then build cloud over
again. It would be easier to just pass the first one along.
Passing along cloud would also make it easier to mock cloud.
When encryptionConfig is enabled, but the secret is missing, there is no
visible errors anywhere. kube-apiserver just goes into a crashloop
without any complains. This PR adds warnings both on the client side and
through nodeup.
When the PublicJWKS feature-flag is set, we expose the apiserver JWKS
document publicly (including enabling anonymous access). This is a
stepping stone to a more hardened configuration where we copy the JWKS
document to S3/GCS/etc.
Co-authored-by: John Gardiner Myers <jgmyers@proofpoint.com>
Application credentials allows you to export a purpose-specific set of
credentials for a user instead of exposing user login credentials.
Especially useful when using LDAP or similar for Openstack users.
Also lets you rotate credentials more easily since multiple application
credentials can be provisioned per user.
Update pkg/model/bootstrapscript.go
Co-authored-by: Ciprian Hacman <ciprianhacman@gmail.com>
Previously as an optimization we would start the kubelet from
protokube, after we had mounted the disks. This helped avoid e.g. the
apiserver going into backoff waiting for etcd.
However, this no longer achieves anything with etcd-manager - nothing
happens on this front until after we start the kubelet anyway.
Doing this both takes protokube out of the dependency sequence here
(slightly faster boot time), but also removes the systemd dependency
from the protokube image. (So we can get a smaller image, perhaps
even distroless)
Errors being fixed or ignored:
```
Errors from staticcheck:
cmd/kops/create_cluster.go:740:37: possible nil pointer dereference (SA5011)
cmd/kops/create_cluster.go:736:7: this check suggests that the pointer can be nil
cmd/kops/create_cluster.go:828:30: possible nil pointer dereference (SA5011)
cmd/kops/create_cluster.go:825:7: this check suggests that the pointer can be nil
dns-controller/pkg/dns/dnscontroller.go:585:5: this value of existing is never used (SA4006)
nodeup/pkg/model/kubelet_test.go:67:23: possible nil pointer dereference (SA5011)
nodeup/pkg/model/kubelet_test.go:63:5: this check suggests that the pointer can be nil
pkg/apis/kops/validation/legacy.go:138:97: unnecessary use of fmt.Sprintf (S1039)
pkg/apis/kops/validation/legacy.go:150:112: unnecessary use of fmt.Sprintf (S1039)
upup/pkg/fi/nodeup/nodetasks/update_packages.go:48:9: unnecessary use of fmt.Sprintf (S1039)
cmd/kops-controller/controllers/node_controller.go:89:1: comment on exported method Reconcile should be of the form "Reconcile ..." (ST1020)
dnsprovider/pkg/dnsprovider/dns.go:102:1: comment on exported function ResourceRecordSetsEquivalent should be of the form "ResourceRecordSetsEquivalent ..." (ST1020)
dnsprovider/pkg/dnsprovider/plugins.go:65:1: comment on exported function RegisteredDnsProviders should be of the form "RegisteredDnsProviders ..." (ST1020)
dnsprovider/pkg/dnsprovider/providers/aws/route53/stubs/route53api.go:30:1: comment on exported type Route53API should be of the form "Route53API ..." (with optional leading article) (ST1021)
dnsprovider/pkg/dnsprovider/providers/google/clouddns/internal/stubs/clouddns.go:26:2: comment on exported type Project should be of the form "Project ..." (with optional leading article) (ST1021)
dnsprovider/pkg/dnsprovider/tests/commontests.go:28:1: comment on exported function CommonTestResourceRecordSetsReplace should be of the form "CommonTestResourceRecordSetsReplace ..." (ST1020)
dnsprovider/pkg/dnsprovider/tests/commontests.go:52:1: comment on exported function CommonTestResourceRecordSetsReplaceAll should be of the form "CommonTestResourceRecordSetsReplaceAll ..." (ST1020)
dnsprovider/pkg/dnsprovider/tests/commontests.go:78:1: comment on exported function CommonTestResourceRecordSetsDifferentTypes should be of the form "CommonTestResourceRecordSetsDifferentTypes ..." (ST1020)
pkg/apis/kops/instancegroup.go:318:1: comment on exported type LoadBalancer should be of the form "LoadBalancer ..." (with optional leading article) (ST1021)
pkg/apis/kops/v1alpha2/instancegroup.go:23:1: comment on exported type InstanceGroup should be of the form "InstanceGroup ..." (with optional leading article) (ST1021)
pkg/apis/kops/v1alpha2/networking.go:449:1: comment on exported type LyftVPCNetworkingSpec should be of the form "LyftVPCNetworkingSpec ..." (with optional leading article) (ST1021)
pkg/dns/gossip.go:21:1: comment on exported function IsGossipHostname should be of the form "IsGossipHostname ..." (ST1020)
pkg/kubeconfig/kubecfg_builder.go:47:1: comment on exported function NewKubeconfigBuilder should be of the form "NewKubeconfigBuilder ..." (ST1020)
pkg/kubeconfig/kubecfg_builder.go:82:1: comment on exported method BuildRestConfig should be of the form "BuildRestConfig ..." (ST1020)
pkg/kubeconfig/kubecfg_builder.go:102:1: comment on exported method WriteKubecfg should be of the form "WriteKubecfg ..." (ST1020)
pkg/model/alimodel/context.go:52:1: comment on exported method LinkToNatGateway should be of the form "LinkToNatGateway ..." (ST1020)
pkg/model/domodel/context.go:21:1: comment on exported type DOModelContext should be of the form "DOModelContext ..." (with optional leading article) (ST1021)
pkg/model/gcemodel/autoscalinggroup.go:38:1: comment on exported type AutoscalingGroupModelBuilder should be of the form "AutoscalingGroupModelBuilder ..." (with optional leading article) (ST1021)
pkg/nodeidentity/do/identify.go:51:1: comment on exported method Token should be of the form "Token ..." (ST1020)
pkg/resources/aws/aws.go:1560:1: comment on exported function ListELBV2s should be of the form "ListELBV2s ..." (ST1020)
pkg/resources/digitalocean/cloud.go:47:1: comment on exported method Token should be of the form "Token ..." (ST1020)
pkg/resources/spotinst/spotinst.go:84:1: comment on exported function NewInstanceGroup should be of the form "NewInstanceGroup ..." (ST1020)
protokube/pkg/gossip/dns/dns.go:29:1: comment on exported const DefaultZoneName should be of the form "DefaultZoneName ..." (ST1022)
protokube/pkg/gossip/mesh/mesh.pb.go:421:4: this value of iNdEx is never used (SA4006)
protokube/pkg/protokube/openstack_volume.go:53:1: comment on exported type OpenstackVolumes should be of the form "OpenstackVolumes ..." (with optional leading article) (ST1021)
upup/pkg/fi/assetstore.go:132:1: comment on exported method AddForTest should be of the form "AddForTest ..." (ST1020)
upup/pkg/fi/cloudup/alitasks/disk.go:128:1: comment on exported method RenderALI should be of the form "RenderALI ..." (ST1020)
upup/pkg/fi/cloudup/alitasks/eip_natgateway_association.go:34:1: comment on exported type EIP should be of the form "EIP ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/alitasks/loadbalancer.go:169:1: comment on exported method RenderALI should be of the form "RenderALI ..." (ST1020)
upup/pkg/fi/cloudup/alitasks/loadbalancerlistener.go:33:1: comment on exported type LoadBalancerListener should be of the form "LoadBalancerListener ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/alitasks/loadbalancerlistener.go:106:1: comment on exported method RenderALI should be of the form "RenderALI ..." (ST1020)
upup/pkg/fi/cloudup/alitasks/natgateway.go:30:1: comment on exported type NatGateway should be of the form "NatGateway ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/alitasks/securitygroup.go:32:1: comment on exported const SecurityResource should be of the form "SecurityResource ..." (ST1022)
upup/pkg/fi/cloudup/alitasks/sshkey.go:33:1: comment on exported type SSHKey should be of the form "SSHKey ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/alitasks/vpc.go:30:1: comment on exported type VPC should be of the form "VPC ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/alitasks/vswitch.go:31:1: comment on exported type VSwitch should be of the form "VSwitch ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/alitasks/vswitchSNAT.go:31:1: comment on exported type VSwitchSNAT should be of the form "VSwitchSNAT ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/aliup/ali_cloud.go:50:1: comment on exported var KubernetesKopsIdentity should be of the form "KubernetesKopsIdentity ..." (ST1022)
upup/pkg/fi/cloudup/awstasks/dhcp_options.go:33:1: comment on exported type DHCPOptions should be of the form "DHCPOptions ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/awstasks/dnsname.go:33:1: comment on exported type DNSName should be of the form "DNSName ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/awstasks/ebsvolume.go:32:1: comment on exported type EBSVolume should be of the form "EBSVolume ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/awstasks/external_load_balancer_attachment.go:31:1: comment on exported type ExternalLoadBalancerAttachment should be of the form "ExternalLoadBalancerAttachment ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/awstasks/external_target_group_attachment.go:32:1: comment on exported type ExternalTargetGroupAttachment should be of the form "ExternalTargetGroupAttachment ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/awstasks/iaminstanceprofile.go:34:1: comment on exported type IAMInstanceProfile should be of the form "IAMInstanceProfile ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/awstasks/iaminstanceprofilerole.go:32:1: comment on exported type IAMInstanceProfileRole should be of the form "IAMInstanceProfileRole ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/awstasks/iamoidcprovider.go:32:1: comment on exported type IAMOIDCProvider should be of the form "IAMOIDCProvider ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/awstasks/iamrole.go:38:1: comment on exported type IAMRole should be of the form "IAMRole ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/awstasks/iamrolepolicy.go:37:1: comment on exported type IAMRolePolicy should be of the form "IAMRolePolicy ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/awstasks/internetgateway.go:30:1: comment on exported type InternetGateway should be of the form "InternetGateway ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/awstasks/launchtemplate_target_cloudformation.go:150:1: comment on exported method CloudformationVersion should be of the form "CloudformationVersion ..." (ST1020)
upup/pkg/fi/cloudup/awstasks/load_balancer.go:39:1: comment on exported type LoadBalancer should be of the form "LoadBalancer ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/awstasks/load_balancer_attachment.go:32:1: comment on exported type LoadBalancerAttachment should be of the form "LoadBalancerAttachment ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/awstasks/natgateway.go:32:1: comment on exported type NatGateway should be of the form "NatGateway ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/awstasks/route.go:31:1: comment on exported type Route should be of the form "Route ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/awstasks/routetable.go:31:1: comment on exported type RouteTable should be of the form "RouteTable ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/awstasks/routetableassociation.go:31:1: comment on exported type RouteTableAssociation should be of the form "RouteTableAssociation ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/awstasks/securitygroup.go:33:1: comment on exported type SecurityGroup should be of the form "SecurityGroup ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/awstasks/securitygrouprule.go:34:1: comment on exported type SecurityGroupRule should be of the form "SecurityGroupRule ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/awstasks/sshkey.go:34:1: comment on exported type SSHKey should be of the form "SSHKey ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/awstasks/subnet.go:32:1: comment on exported type Subnet should be of the form "Subnet ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/awstasks/vpc.go:33:1: comment on exported type VPC should be of the form "VPC ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/awstasks/vpc_dhcpoptions_association.go:30:1: comment on exported type VPCDHCPOptionsAssociation should be of the form "VPCDHCPOptionsAssociation ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/awstasks/vpccidrblock.go:29:1: comment on exported type VPCCIDRBlock should be of the form "VPCCIDRBlock ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/awsup/aws_cloud.go:58:1: comment on exported const ClientMaxRetries should be of the form "ClientMaxRetries ..." (ST1022)
upup/pkg/fi/cloudup/awsup/status.go:44:1: comment on exported method FindClusterStatus should be of the form "FindClusterStatus ..." (ST1020)
upup/pkg/fi/cloudup/dotasks/droplet.go:32:1: comment on exported type Droplet should be of the form "Droplet ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/dotasks/loadbalancer.go:35:1: comment on exported type LoadBalancer should be of the form "LoadBalancer ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/dotasks/volume.go:32:1: comment on exported type Volume should be of the form "Volume ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/gcetasks/address.go:29:1: comment on exported type Address should be of the form "Address ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/gcetasks/instance.go:33:1: comment on exported type Instance should be of the form "Instance ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/gcetasks/instancegroupmanager.go:29:1: comment on exported type InstanceGroupManager should be of the form "InstanceGroupManager ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/gcetasks/instancetemplate.go:35:1: comment on exported const InstanceTemplateNamePrefixMaxLength should be of the form "InstanceTemplateNamePrefixMaxLength ..." (ST1022)
upup/pkg/fi/cloudup/gcetasks/network.go:30:1: comment on exported type Network should be of the form "Network ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/gcetasks/subnet.go:30:1: comment on exported type Subnet should be of the form "Subnet ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/openstacktasks/floatingip.go:32:1: comment on exported type FloatingIP should be of the form "FloatingIP ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/openstacktasks/instance.go:32:1: comment on exported type Instance should be of the form "Instance ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/openstacktasks/lb.go:34:1: comment on exported type LB should be of the form "LB ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/openstacktasks/lblistener.go:30:1: comment on exported type LBListener should be of the form "LBListener ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/openstacktasks/lbpool.go:28:1: comment on exported type LBPool should be of the form "LBPool ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/openstacktasks/network.go:28:1: comment on exported type Network should be of the form "Network ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/openstacktasks/poolassociation.go:30:1: comment on exported type PoolAssociation should be of the form "PoolAssociation ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/openstacktasks/port.go:29:1: comment on exported type Port should be of the form "Port ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/openstacktasks/router.go:28:1: comment on exported type Router should be of the form "Router ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/openstacktasks/routerinterface.go:29:1: comment on exported type RouterInterface should be of the form "RouterInterface ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/openstacktasks/securitygroup.go:31:1: comment on exported type SecurityGroup should be of the form "SecurityGroup ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/openstacktasks/servergroup.go:30:1: comment on exported type ServerGroup should be of the form "ServerGroup ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/openstacktasks/sshkey.go:31:1: comment on exported type SSHKey should be of the form "SSHKey ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/openstacktasks/subnet.go:29:1: comment on exported type Subnet should be of the form "Subnet ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/openstacktasks/volume.go:28:1: comment on exported type Volume should be of the form "Volume ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/spotinsttasks/elastigroup.go:42:1: comment on exported type Elastigroup should be of the form "Elastigroup ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/spotinsttasks/launch_spec.go:37:1: comment on exported type LaunchSpec should be of the form "LaunchSpec ..." (with optional leading article) (ST1021)
upup/pkg/fi/cloudup/spotinsttasks/ocean.go:39:1: comment on exported type Ocean should be of the form "Ocean ..." (with optional leading article) (ST1021)
upup/pkg/fi/context.go:249:1: comment on exported function NewExistsAndWarnIfChangesError should be of the form "NewExistsAndWarnIfChangesError ..." (ST1020)
upup/pkg/fi/context.go:256:1: comment on exported method Error should be of the form "Error ..." (ST1020)
upup/pkg/fi/fitasks/keypair.go:31:1: comment on exported type Keypair should be of the form "Keypair ..." (with optional leading article) (ST1021)
upup/pkg/fi/fitasks/keypair.go:55:1: comment on exported method CheckExisting should be of the form "CheckExisting ..." (ST1020)
upup/pkg/fi/fitasks/managedfile.go:29:1: comment on exported type ManagedFile should be of the form "ManagedFile ..." (with optional leading article) (ST1021)
upup/pkg/fi/fitasks/mirrorkeystore.go:25:1: comment on exported type MirrorKeystore should be of the form "MirrorKeystore ..." (with optional leading article) (ST1021)
upup/pkg/fi/fitasks/mirrorsecrets.go:26:1: comment on exported type MirrorSecrets should be of the form "MirrorSecrets ..." (with optional leading article) (ST1021)
upup/pkg/fi/fitasks/secret.go:25:1: comment on exported type Secret should be of the form "Secret ..." (with optional leading article) (ST1021)
upup/pkg/fi/fitasks/secret.go:33:1: comment on exported method CheckExisting should be of the form "CheckExisting ..." (ST1020)
upup/pkg/fi/resources.go:248:1: comment on exported method AsBytes should be of the form "AsBytes ..." (ST1020)
upup/pkg/kutil/import_cluster.go:680:1: comment on exported function GetInstanceUserData should be of the form "GetInstanceUserData ..." (ST1020)
```
Ole Markus With
Ole Markus With
Ciprian Hacman
John Gardiner Myers
Kenji Kaneda
Peter Rifel
Barry Melbourne
Kubernetes Prow Robot
Bharath Vedartham
Steven E. Harris
Justin SB
shil
Ciprian Hacman
Rodrigo Menezes
AkiraFukushima