diff --git a/archive/wg-component-standard/component-config/README.md b/archive/wg-component-standard/component-config/README.md new file mode 100644 index 000000000..66e7185bc --- /dev/null +++ b/archive/wg-component-standard/component-config/README.md @@ -0,0 +1,980 @@ +This copy of [Versioned Component Configuration Files](vccf-proposal) +was automatically converted from Google Docs to Markdown +so that it could be included in the community archive. +Several interesting comment threads remain unexported but +available in the Google Doc. You can gain access to the +doc by joining the `dev@kubernetes.io` mailing list +(previously `kubernetes-dev@googlegroups.com`). Please +do not request access via the Google Docs UI, as this +spams the owners with access requests. Thank you :). + +[vccf-proposal]: https://docs.google.com/document/d/1FdaEJUEh091qf5B98HM6_8MS764iXrxxigNIdwHYW9c/ + + +# Versioned Component Configuration Files + + +## How Kubelet learned to stop using flags and love versioned config. How your component can, too. + +**Shared publicly!** + +**Author:** mtaufen@google.com + +**Last Updated:** March 28, 2018 + +**Self Link:** https://goo.gl/GM8KyH + +# Background + +A long time ago (but still in this galaxy), someone had the bright idea that we could avoid a lot of the pain of deploying and managing command-line flags for each core cluster component (kubelet, kube-proxy, scheduler, etc.) by switching to Kubernetes-style versioned configuration files. This effort became known to the community as _component configuration_, or simply _componentconfig_. At its origin, a consistent philosophy for _what_ componentconfig should look like did not exist. + +Last year, mikedanese@ did a great job of compiling the ideas behind componentconfig (ideas Mike, other folks, and I were discussing in several GitHub threads) into a single [document](https://docs.google.com/document/d/1arP4T9Qkp2SovlJZ_y790sBeiWXDO6SG10pZ_UUU-Lc), in the hope that we could provide standard guidelines and improve consistency across the project. Mike's document catalyzed our push to try componentconfig in a few components, and here we are almost a year later. + +As of Kubernetes v1.10, the Kubelet is firmly on its way to migrating from flags to versioned configuration files. It can consume a beta-versioned config file and many flags are now deprecated and pending removal in favor of this file. Many remaining flags will be replaced by the file over time. Additionally, the kube-proxy component is very close to having a beta-versioned config file of its own. + +This document restates the motivation and records lessons from OSS work over the past year: + + + +* a brief review of why we want versioned config files for all core cluster components +* the _ideal_ state of a component's configuration API +* how to migrate an existing component to versioned configuration files (Kubelet example) +* remaining work + + +# Why versioned config files? + +The short answer is that flags are nonstandard interfaces with weak stability guarantees. They are confusing and hard to deploy, and this is the opposite of what Kubernetes should be. + +Command-line flags present a number of problems: + + + +* Flags are a public API, but are not versioned and cannot be versioned separately from the binary: + * For _core_ components the binary version is coupled to the Kubernetes release version. We use semantic versioning for our binaries, but can't bump a major version unless Kubernetes does: + * We _shouldn't_ ever **fix bad defaults** for existing flag names without bumping the major version of a binary. In reality, we get around this by giving advance warning and technically breaking semantic versioning of the binary. + * We _shouldn't_ ever **remove a flag** without bumping the major version of a binary. In reality we use a [flag deprecation policy](https://kubernetes.io/docs/reference/deprecation-policy/#deprecating-a-flag-or-cli) that allows us to technically break semantic versioning of the binary as long as we give advance warning. + * We incrementally deprecate individual parameters over time, instead of guaranteeing a consistent set of parameters for the life of an API version. This confuses users and results in a less stable API. + * We can't typically deploy flag-based configuration independently from a binary version upgrade, because the compatibility of the interface is so tightly coupled. +* Values are often re-configured, which precipitates additional tools to parameterize and write configuration for system-specific process management agents (e.g. systemd). We can eliminate the dependency on parameterization tools if process manager config is static; e.g. the configured command line just needs to reference a file in a fixed location. +* Developers inevitably embed structured data in strings and invent one-off parsers to process their flags. This invites bugs. +* mikedanese@ outlined more issues in his [document](https://docs.google.com/document/d/1arP4T9Qkp2SovlJZ_y790sBeiWXDO6SG10pZ_UUU-Lc). + +Core goals of componentconfig include: + + + +* Standardize the configuration approach for all core cluster components. +* Enable dynamic configuration deployment mechanisms. + +Conveniently, Kubernetes has similar goals: + + + +* Standardize the configuration approach for cluster infrastructure. +* Enable dynamic deployment mechanisms. + +Kubernetes had already paved the way: It has what we need to version our configuration interfaces, decouple configuration changes from binary changes, represent configuration in a structured format, and deploy configuration in a dynamic environment. + + + +* Versioning was accomplished via the API machinery's group/version mechanism. +* Adhering to the same API guarantees as the core Kubernetes APIs provides a stable configuration surface, and allows us to decouple the configuration interface from releases that support the same API version. +* Kubernetes API objects are defined as Go structs, which means we don't have to parse strings to deal with structured config. +* Kubernetes has deployment mechanisms (ConfigMap volume source) that work well for pushing new versions of configuration files into production. +* There is no requirement to restart a process when you change a file, unlike flags. + +**All core Kubernetes components should eventually consume their configuration via versioned configuration files, instead of command-line flags.** + + +# tl;dr: What should a component do? + +**This is the _ideal_ command-line API for every core cluster component:** + +```console +$ component --config=path +``` + +The component exposes _only one_ flag on its command line. This flag provides the file path to a config file with a versioned format. All other relevant configuration information is referenced via this file. + +_One, stable flag where everything else is versioned config_ is the ideal API recommended by componentconfig. If you are creating a new component from scratch, begin and end with this API. + +For several reasons discussed in the next section, the migration from flags to versioned config is a serious journey for most existing components, and these may want or even need a couple more flags at the end of the day. + +In general, every core cluster component should: + + + +* Maintain a distinct Kubernetes API group called {component}.config.k8s.io, which contains versioned sets of config objects - primarily a {Component}Configuration struct in each version. This struct, serialized to disk by the API machinery, is the file format for configuration. +* Ensure {component}.config.k8s.io adheres to the standard Kubernetes [API deprecation policy](https://kubernetes.io/docs/reference/deprecation-policy/#deprecating-parts-of-the-api), [API conventions](https://github.com/kubernetes/community/blob/master/contributors/devel/api-conventions.md), and [API changes policy](https://github.com/kubernetes/community/blob/master/contributors/devel/api_changes.md). +* Expose a flag named --config, which accepts a path to a file that contains a serialized {Component}Configuration struct. +* Use the Kubernetes API machinery to deserialize the config file data, apply defaults, and convert to an internal version for runtime use. +* Validate the internal version prior to using it. If validation fails, refuse to run with the specified configuration. +* Ensure third-party libraries aren't leaking flags. + +We discuss how to migrate a component from flags to versioned config files in the next section. + + +# In-depth: How to migrate a component from flags to a versioned config file + + +## Take back control of the command-line API + +Our goal is to _decrease_ flag usage in favor of versioned config files. It will help to decrease the growth rate of the component's flag API. There are at least two sources of this growth: + + + +* PRs that directly extend the flag API. +* Adding or updating third-party libraries. + +Whoever owns the componentconfig effort for a given component should be in-the-loop on PRs that add new flags. This person has a strong interest in saying "no" to new flags, because it increases the number of things they have to carefully migrate to the new versioned config file API. When new flags really prove necessary, this person still has a strong interest in ensuring they will be compatible when migrated. + +The second case is an artifact of many libraries registering flags globally (global flag sets are provided by both the _flag_ and _pflag_ libraries). Since most components just parse the global flag sets by default, they tend to accumulate the flags from these libraries. The libraries are impolite, and the components are typically too trusting. Each component should be more cautious by: + + + +* constructing its own, isolated flag set +* explicitly registering necessary flags from third-party libraries into this flag set +* parsing _only_ the flags in this flag set + +You can find the example of how the Kubelet took back control of its flag set in [Explicit kubelet flags](https://github.com/kubernetes/kubernetes/pull/57613) (see also the follow-up PR, [#58095](https://github.com/kubernetes/kubernetes/pull/58095)). + +Many components indiscriminately add the global flag set to their primary flag set via `pflag.CommandLine.AddGoFlagSet(flag.CommandLine)`. Further, most components delegate their flag parsing, help text generation, etc. to [Cobra](https://github.com/spf13/cobra). Cobra implicitly adds flag.CommandLine in several cases, which unfortunately gets in the way of explicit control over the flag API. + +In order for Cobra to parse flags for you, it has to be made aware of your flagset. This is achieved by registering flags to the command's flag set. Cobra will implicitly merge the global command lines with this flag set when it parses flags. Consider the following Go program, which creates a local, isolated flag set (as recommended above). The program also registers a global flag, which we _hope_ won't be parsed, because it is not explicitly registered with our local flag set. + + +```go +package main + +import ( + "fmt" + "os" + + "github.com/spf13/cobra" + "github.com/spf13/pflag" +) + +const use = "testcmd" + +var ( + globalFlagTarget string + localFlagTarget string +) + +func init() { + pflag.StringVar(&globalFlagTarget, "global-flag", globalFlagTarget, "globally-registered flag") +} + +func NewLocalFlagSet() *pflag.FlagSet { + fs := pflag.NewFlagSet(use, pflag.ContinueOnError) + fs.StringVar(&localFlagTarget, "local-flag", localFlagTarget, "locally-registered flag") + return fs +} + +func NewTestCmd() *cobra.Command { + cmd := &cobra.Command{ + Use: use, + Run: func(cmd *cobra.Command, args []string) { + fmt.Printf("globalFlagTarget: %q\n", globalFlagTarget) + fmt.Printf("localFlagTarget: %q\n", localFlagTarget) + }, + } + cmd.Flags().AddFlagSet(NewLocalFlagSet()) + return cmd +} + +func main() { + cmd := NewTestCmd() + if err := cmd.Execute(); err != nil { + fmt.Fprintf(os.Stderr, "error: %v\n", err) + os.Exit(1) + } +} +``` + + +If we run the program, we see that this is not the case. The global flag is parsed by Cobra: + + +```console +$ testcmd --global-flag hello + +globalFlagTarget: "hello" +localFlagTarget: "" +``` + + +You can circumvent this by disabling Cobra's flag parsing. This, unfortunately, requires that you both parse flags and implement --help short-circuiting on your own. The next example attempts to extend the above to do so. + + +```go +package main + +import ( + "fmt" + "os" + + "github.com/spf13/cobra" + "github.com/spf13/pflag" +) + +const use = "testcmd" + +var ( + globalFlagTarget string + localFlagTarget string +) + +func init() { + pflag.StringVar(&globalFlagTarget, "global-flag", globalFlagTarget, "globally-registered flag") +} + +func NewLocalFlagSet() *pflag.FlagSet { + fs := pflag.NewFlagSet(use, pflag.ContinueOnError) + fs.StringVar(&localFlagTarget, "local-flag", localFlagTarget, "locally-registered flag") + return fs +} + +func NewTestCmd() *cobra.Command { + localFlagSet := NewLocalFlagSet() + cmd := &cobra.Command{ + Use: use, + DisableFlagParsing: true, + Run: func(cmd *cobra.Command, args []string) { + // parse our local flag set + if err := localFlagSet.Parse(args); err != nil { + cmd.Usage() + fatal(err) + } + // --help + help, err := localFlagSet.GetBool("help") + if err != nil { + fatal(fmt.Errorf(`"help" flag is non-bool, programmer error, please correct`)) + } + if help { + cmd.Help() + return + } + // print the flag values + fmt.Printf("globalFlagTarget: %q\n", globalFlagTarget) + fmt.Printf("localFlagTarget: %q\n", localFlagTarget) + }, + } + localFlagSet.BoolP("help", "h", false, fmt.Sprintf("help for %s", cmd.Name())) + // Cobra still needs to be aware of our flag set to generate usage and help text + cmd.Flags().AddFlagSet(localFlagSet) + return cmd +} + +func fatal(err error) { + fmt.Fprintf(os.Stderr, "error: %v\n", err) + os.Exit(1) +} + +func main() { + cmd := NewTestCmd() + if err := cmd.Execute(); err != nil { + fatal(err) + } +} +``` + + +When we run the program and attempt to set the global flag, we see that the global flag is now rejected, but we also see that it is still included in the usage text! + + +```console +$ testcmd --global-flag hello +Usage: + testcmd [flags] + +Flags: + --global-flag string globally-registered flag + -h, --help help for testcmd + --local-flag string locally-registered flag +error: unknown flag: --global-flag +``` + + +The same thing happens when we pass --help: + + +```console +$ testcmd --help +Usage: + testcmd [flags] + +Flags: + --global-flag string globally-registered flag + -h, --help help for testcmd + --local-flag string locally-registered flag +``` + + +This is because Cobra also uses the global flags when generating usage and help text. This can be circumvented by doing-it-yourself, again. + + +```go +package main + +import ( + "fmt" + "os" + + "github.com/spf13/cobra" + "github.com/spf13/pflag" +) + +const use = "testcmd" + +var ( + globalFlagTarget string + localFlagTarget string +) + +func init() { + pflag.StringVar(&globalFlagTarget, "global-flag", globalFlagTarget, "globally-registered flag") +} + +func NewLocalFlagSet() *pflag.FlagSet { + fs := pflag.NewFlagSet(use, pflag.ContinueOnError) + fs.StringVar(&localFlagTarget, "local-flag", localFlagTarget, "locally-registered flag") + return fs +} + +func NewTestCmd() *cobra.Command { + localFlagSet := NewLocalFlagSet() + cmd := &cobra.Command{ + Use: use, + DisableFlagParsing: true, + Run: func(cmd *cobra.Command, args []string) { + // parse our local flag set + if err := localFlagSet.Parse(args); err != nil { + cmd.Usage() + fatal(err) + } + // --help + help, err := localFlagSet.GetBool("help") + if err != nil { + fatal(fmt.Errorf(`"help" flag is non-bool, programmer error, please correct`)) + } + if help { + cmd.Help() + return + } + // print the flag values + fmt.Printf("globalFlagTarget: %q\n", globalFlagTarget) + fmt.Printf("localFlagTarget: %q\n", localFlagTarget) + }, + } + localFlagSet.BoolP("help", "h", false, fmt.Sprintf("help for %s", cmd.Name())) + + // ugly, but necessary, because Cobra's default UsageFunc and HelpFunc pollute the flagset with global flags + const usageFmt = "Usage:\n %s\n\nFlags:\n%s" + cmd.SetUsageFunc(func(cmd *cobra.Command) error { + fmt.Fprintf(cmd.OutOrStderr(), usageFmt, cmd.UseLine(), localFlagSet.FlagUsagesWrapped(2)) + return nil + }) + cmd.SetHelpFunc(func(cmd *cobra.Command, args []string) { + fmt.Fprintf(cmd.OutOrStdout(), usageFmt, cmd.UseLine(), localFlagSet.FlagUsagesWrapped(2)) + }) + + return cmd +} + +func fatal(err error) { + fmt.Fprintf(os.Stderr, "error: %v\n", err) + os.Exit(1) +} + +func main() { + cmd := NewTestCmd() + if err := cmd.Execute(); err != nil { + fatal(err) + } +} +``` + + +And now things are more as we expect. + + +```console +$ testcmd --global-flag hello +Usage: + testcmd [flags] + +Flags: + -h, --help help for testcmd + --local-flag string locally-registered flag +error: unknown flag: --global-flag + +$ testcmd --help +Usage: + testcmd [flags] + +Flags: + -h, --help help for testcmd + --local-flag string locally-registered flag +``` + + +**Alternative solutions to piecewise-DIY-overrides are highly welcomed** (if someone wants to write a Cobra replacement that meets our needs while managing state more cleanly, I won't stop you). As all core components will need to solve this one way or another, a centralized utility library for working with Cobra would be useful, at the very least. + + +### Use a flags struct + +The migration from flags to versioned config files will be much easier if you first centralize where your target flag values, registrations, and deprecations happen. If you have a single structure definition that contains all of the component's flag-targeted values, you can focus on moving fields from this structure into your versioned configuration API. + +The Kubelet uses a structure called KubeletFlags, with an associated `func (f *KubeletFlags) AddFlags(fs *pflag.FlagSet)` to handle flag registrations and deprecations. Note that AddFlags does not register global flags from third-party libraries; it is only concerned with flags in the KubeletFlags structure. + +We also recommend that defaulting behavior for flags be clearly separated from the flag registrations. Kubelet initializes flag defaults when constructing a new KubeletFlags, and re-uses these values when registering flags. This makes it easy to see which defaults are applied, which makes it easy to migrate those defaults to versioned config. This also prevents AddFlags from overriding values in the flags struct, in the event that you need to modify values before registering flags. + +Finally, the Kubelet offers a function for validating the flags structure. You may choose to centralize validation here, which will make it easier to migrate that validation to your versioned config, and will also elevate configuration errors to sooner in the component's lifecycle. **It is important to point out that burying flag validation in application logic is an anti-pattern that should be avoided whenever possible. Given the opportunity, components should be refactored to centralize validation.** The Kubelet unfortunately falls into the "validation all over the place" trap, and will eventually need to be refactored to centralize validation. + +KubeletFlags currently contains some flags that are only registered on specific operating systems (e.g. Windows). These fields are prefixed with the name of the OS (e.g. `Windows*`) and registrations are handled by OS-specific implementations of the addOSFlags method (managed via Go build tags). + +This excerpt from the Kubelet's flag code gives the general structure: + + +
cmd/kubelet/app/options/options.go + | +
type KubeletFlags struct {
+ + KubeletConfigFile string + + … + +} + +func NewKubeletFlags() *KubeletFlags { + + return &KubeletFlags{ + + // apply defaults here + + } + +} + +func (f *KubeletFlags) AddFlags(fs *pflag.FlagSet) { + + f.addOSFlags(fs) + + fs.StringVar(&f.KubeletConfigFile, "config", f.KubeletConfigFile, "…") + + … + +} + +func ValidateKubeletFlags(f *KubeletFlags) error { + + // validate here, return error if validation fails, nil otherwise + + … + +} + |
+
cmd/kubelet/app/options/osflags_windows.go + | +
// +build windows
+ +… + +func (f *KubeletFlags) addOSFlags(fs *pflag.FlagSet) { + + // add windows flags here + +} + |
+
cmd/kubelet/app/options/osflags_other.go + | +
// +build !windows
+ +… + +func (f *KubeletFlags) addOSFlags(fs *pflag.FlagSet) { + + // noop + +} + |
+
cmd/kubelet/app/options/options.go + | +
func NewKubeletConfiguration() (*kubeletconfig.KubeletConfiguration, error) {
+ + scheme, _, err := kubeletscheme.NewSchemeAndCodecs() + + if err != nil { + + return nil, err + + } + + versioned := &v1beta1.KubeletConfiguration{} + + scheme.Default(versioned) + + config := &kubeletconfig.KubeletConfiguration{} + + if err := scheme.Convert(versioned, config, nil); err != nil { + + return nil, err + + } + + applyLegacyDefaults(config) + + return config, nil + +} + +func AddKubeletConfigFlags(fs *pflag.FlagSet, c *kubeletconfig.KubeletConfiguration) { + + // register flags here, in the same style as in KubeletFlags.AddFlags + + … + +} + |
+
pkg/kubelet/kubeletconfig/util/codec/codec.go + | +
// DecodeKubeletConfiguration decodes a serialized KubeletConfiguration to the internal type
+ +func DecodeKubeletConfiguration(kubeletCodecs *serializer.CodecFactory, data []byte) (*kubeletconfig.KubeletConfiguration, error) { + + // the UniversalDecoder runs defaulting and returns the internal type by default + + obj, gvk, err := kubeletCodecs.UniversalDecoder().Decode(data, nil, nil) + + if err != nil { + + return nil, fmt.Errorf("failed to decode, error: %v", err) + + } + + internalKC, ok := obj.(*kubeletconfig.KubeletConfiguration) + + if !ok { + + return nil, fmt.Errorf("failed to cast object to KubeletConfiguration, unexpected type: %v", gvk) + + } + + return internalKC, nil + +} + |
+
pkg/kubelet/apis/kubeletconfig/helpers.go + | +
// KubeletConfigurationPathRefs returns pointers to all of the KubeletConfiguration fields that contain filepaths.
+ +// You might use this, for example, to resolve all relative paths against some common root before + +// passing the configuration to the application. This method must be kept up to date as new fields are added. + +func KubeletConfigurationPathRefs(kc *KubeletConfiguration) []*string { + + paths := []*string{} + + paths = append(paths, &kc.StaticPodPath) + + paths = append(paths, &kc.Authentication.X509.ClientCAFile) + + paths = append(paths, &kc.TLSCertFile) + + paths = append(paths, &kc.TLSPrivateKeyFile) + + paths = append(paths, &kc.ResolverConfig) + + return paths + +} + |
+
pkg/kubelet/kubeletconfig/configfiles/configfiles.go + | +
// resolveRelativePaths makes relative paths absolute by resolving them against `root`
+ +func resolveRelativePaths(paths []*string, root string) { + + for _, path := range paths { + + // leave empty paths alone, "no path" is a valid input + + // do not attempt to resolve paths that are already absolute + + if len(*path) > 0 && !filepath.IsAbs(*path) { + + *path = filepath.Join(root, *path) + + } + + } + +} + |
+
cmd/kubelet/app/server.go + | +
// newFakeFlagSet constructs a pflag.FlagSet with the same flags as fs, but where
+ +// all values have noop Set implementations + +func newFakeFlagSet(fs *pflag.FlagSet) *pflag.FlagSet { + + ret := pflag.NewFlagSet("", pflag.ExitOnError) + + ret.SetNormalizeFunc(fs.GetNormalizeFunc()) + + fs.VisitAll(func(f *pflag.Flag) { + + ret.VarP(flag.NoOp{}, f.Name, f.Shorthand, f.Usage) + + }) + + return ret + +} + |
+
cmd/kubelet/app/server.go + | +
// Remember original feature gates, so we can merge with flag gates later
+ +original := kc.FeatureGates + +// re-parse flags + +if err := fs.Parse(args); err != nil { + + return err + +} + +// Add back feature gates that were set in the original kc, but not in flags + +for k, v := range original { + + if _, ok := kc.FeatureGates[k]; !ok { + + kc.FeatureGates[k] = v + + } + +} + |
+