kubernetes
/
community
mirror of https://github.com/kubernetes/community.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Move the doc to new location and remove scratch section

This commit is contained in:
Mike Arpaia 2018-04-10 18:15:15 -06:00
parent 3d7fbe342b
commit f4097d40f3
No known key found for this signature in database
GPG Key ID: 257B170C423DE4A0
1 changed files with 0 additions and 190 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

190
wg-apply/declarative-application-management.md → contributors/design-proposals/architecture/declarative-application-management.md
View File

@ -393,193 +393,3 @@ Deployment pipelines, [canary deployments](https://groups.google.com/forum/#!top
#### What about UI wizards, IDE integration, application frameworks, etc.?
Representing configuration using the literal API types should facilitate programmatic manipulation of the configuration via user-friendly tools, such as UI wizards (e.g., [dashboard](https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/#deploying-containerized-applications), [Yipee.io](https://yipee.io/), and many CD tools, such as [Distelli](https://www.distelli.com/docs/k8s/add-container-to-a-project/)) and IDEs (e.g., [VSCode](https://www.youtube.com/watch?v=QfqS9OSVWGs), [IntelliJ](https://github.com/tinselspoon/intellij-kubernetes)), as well as configuration generation and manipulation by application frameworks (e.g., [Spring Cloud](https://github.com/fabric8io/spring-cloud-kubernetes)).
## Application Definition Format Sketch
This (still half-baked) proposal includes solutions to several practical problems with relatively modest extensions to existing kubectl capabilities.
The vast majority of substitutions are names and labels. After that, images/tags, replicas, cpu/memory resources, local vs. remote disk (hopefully will be simplified by local PVC support), and application configuration. Within Google, application configuration (e.g., setting command-line flags) accounts for most lines of Borg configuration.
Basic idea:
* Use literal API resources only
* Encourage forking/copying and local modification as the primary means of customization
* The goal is for "kubectl apply -f packagedir/" to transform and declaratively reconcile all of the resources for an application.
* Building tooling to declaratively customize resource names and labels, including embedded selectors, pod template labels, and object references. (My conclusion from playing with this for a little while is that this is the only thing hard today. If we built a tool to do that, a POC could be built using a shell script.)
* Use an overlay model to instantiate multiple variants instead of parameterization: Do the equivalent of "kubectl patch" to apply files from one directory as patches to ones in another directory.
* Explanation: While overlays are needed in order to support multiple variants, they are inherently more complex to reason about than unpatched resource definitions that can be achieved via forking. Additionally, overlays introduce complexity into the upgrade procedure, in the case that the underlying resource changes in ways that are no longer compatible with the overlays. Forking facilitates a standard version-control-based rebase/merge, review, and conflict-resolution workflow.
* Create a way to declaratively create new configmaps and secrets from source data.
* Make it easy to update images and configmap and secret references, since those are frequent changes that need to be rolled out.
* Build more tooling to generate and edit config files, as well as to copy them around, fork and rebase them, etc.
* Better object selection support would be useful, to apply changes to a subset of objects or to select one of a number of alternatives (rather than embedded conditionals).
Additional tooling would be needed to automate common customization patterns, such as the [environment customization required by our github bot](https://github.com/kubernetes/test-infra/tree/master/charts/mungegithub/chart). My current preferred solution is to use (single-layer) overlays. The "package" would contain package metadata (e.g., description, version, source URL, maintainers) and base resources, perhaps including optional ones. Instance directories would contain instantiation instructions (e.g., name prefixes and labels to inject), resource overlays (strategic merge patches), and additional resources, such as ConfigMaps containing application configuration and HorizontalPodAutoscaler resources.
One possible user workflow would be:
1. Discover a package you want to use via some kind of registry
2. Pull it and install it locally
3. Use tools to mutate it: kubectl set foo, kubectl edit, vscode, etc.
4. If desired, use tools to set up a fork and commit it
5. Use tools to create a set of overlays in another directory.
6. Commit those overlays, if desired
7. Automate name+label customization (TODO: describe prefixing of references to external resources) and patching of the overlays onto the forked resources, then kubectl apply. The result could be output and checked in, if desired. This is equivalent to pairing up resources and doing "kubectl patch --local -o yaml ... | munge-names-and-labels | kubectl apply -f -"
8. Auto-deploy from checked-in configs, if desired.
The overlay patch step is only really necessary when the user may want to deploy multiple customized instances. I’d recommend forking over overlays for base customization, in order to make use of standard version-control tools for integrating upstream changes to the base resources. Changes in the base could still conflict with any overlays, but a simple convention such as specifying default values for fields overridden by overlays would make that easier to detect.
Both the fork and overlay approaches would make it easy to add additional resources to a base set of resources, such as secrets, HorizontalPodAutoscaler, vertical autoscaling, PodPreset, service bindings, etc.
I do not think we should support arbitrary numbers of stacked overlays. An internal project attempted that before, and was cancelled. At least one of the problems is similar to the issue with docker image layers, where changing a base layer invalidates layers stacked on top of it. Beyond a couple layers it also becomes hard to reason about the implications of each overlay. In general, I expect the user to fork the package, customize it, and then write overlays. An update of the package would require rebasing and resolving conflicts if they made any changes. In reality, that's typically necessary when using parameterization, also.
However, we do need to support alternative subsets of resources, which may be easier if they are in separate directories, rather than relying only on object selectors. We should evaluate the tradeoffs between simply forking to select an alternative (e.g., commenting/uncommenting alternative lines), specifying selected subdirectories (which may require more object duplication), forking provided overlays (to both select among alternatives and instantiate multiple variants), and supporting multiple layers of overlays (which would be the most automated solution but would also have the most complex behavior).
I'm using the [mungegithub chart](https://github.com/kubernetes/test-infra/tree/master/charts/mungegithub/chart) as an example (to eat our own dogfood; TODO: also look at the [Spinnaker chart](https://github.com/kubernetes/charts/tree/master/stable/spinnaker)). mungegithub is like most applications: it doesn't actually need ANY parameters. It just needs name and label customization, the ability to add resources, and the ability to set the image.
The template/package directory would contain literal resource files for the service, deployment, and pvc, and also package metadata (package description, version, source, approvers). Outside that directory would be one instance-specific subdirectory per customized instance (kubernetes, kubernetes.github.io, test-infra, contrib). Each of those would contain instance data (e.g. name prefixes and labels to inject), deployment overlay, and configmap. If it's easier, an empty configmap could reside in the package directory. FWIW, that's similar to an add-on use case.
One can experiment with this approach by doing the following:
```
kubectl patch --local -f package/deployment.yaml \
--patch="$(< instances/contrib/deployment.yaml)" -o yaml
```
where the overlay in instances/contrib/deployment.yaml is:
```
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: mungegithub
spec:
template:
spec:
containers:
- name: submit-queue
volumeMounts:
- mountPath: /etc/munge-config
name: munge-config
volumes:
- name: munge-config
configMap:
name: mungegithub-config
```
The full base resource spec is long, so it is omitted here, but the result is a merge of the two partial resource specifications.
This was just an example. The deployment template in the helm chart is written to expect secrets to be provided. There's no reason why it couldn't expect configmaps to be provided, also.
Initial stab at manifest format.
Base resource ("package") Kube-manifest.yaml file (name TBD, but it needs to be a well known file name in order to work with bulk/recursive traversal):
```
# inspired by https://github.com/kubernetes/helm/blob/master/docs/charts.md
# but Kubernetes API style:
apiVersion: manifest.k8s.io/v1alpha1
Kind: Package # was BaseManifest
metadata:
name: mungebot
description: Mungegithub package
appVersion: 0.1.0
home: https://github.com/bgrant0607/mungebot-pkg/blob/master/README.md
sources:
- https://github.com/bgrant0607/mungebot-pkg.git
icon: https://github.com/bgrant0607/mungebot-pkg/blob/master/icon.png
maintainers:
- name: Brian Grant
- email: briangrant@google.com
- github: bgrant0607
resources:
- service.yaml
- deployment.yaml
- pvc.yaml
# Recursive would be similar to kubectl --recursive behavior, extended to look for Kube-manifest.yaml
recursive: false
# whether PersistentVolumeClaims should be deleted with the other resources
ownPersistentVolumeClaims: true
# Directory should also contain:
#
# - README.txt or README.md
# - OWNERS
# - LICENSE
```
Overlay ("instance") Kube-manifest.yaml file:
```
apiVersion: manifest.k8s.io/v1alpha1
kind: Package # was OverlayManifest
metadata:
name: test-infra-mungebot
description: Mungebot config for test-infra repo
namePrefix: test-infra-
# Labels to add to all objects and selectors.
# These labels would also be used to form the selector for apply --prune
# Named differently than "labels" to avoid confusion with metadata for this object
objectLabels:
app: mungebot
org: kubernetes
repo: test-infra
objectAnnotations:
note: This is my first try
resources:
- ../../package
# These are strategic merge patch overlays in the form of API resources
patches:
- deployment.yaml
# There could also be configmaps in Base, which would make these overlays
configmaps:
- type: env
namePrefix: app-env
file: app.env
- type: file
namePrefix: app-config
file: app-init.ini
# There could be secrets in Base, if just using a fork/rebase workflow
secrets:
- type: tls
namePrefix: app-tls
certFile: tls.cert
keyFile: tls.key
recursive: false
prune: true # I’d make this the default
```
Both the secrets and configmaps specify name prefixes rather than complete resource names. Their name suffixes should be deterministic hashes of their contents. We need to be able to update references to them similar to name prefixing.
The directory structure I'm imagining is:
- ./package/LICENSE
- ./package/OWNERS
- ./package/README.md
- ./package/icon.png
- ./package/Kube-manifest.yaml
- ./package/service.yaml
- ./package/deployment.yaml
- ./package/pvc.yaml
- ./instances/test-infra/Kube-manifest.yaml
- ./instances/test-infra/deployment.yaml
- ./instances/test-infra/app.env
- ./instances/test-infra/app-init.ini
- ./instances/test-infra/tls.cert
- ./instances/test-infra/tls.key
- ./instances/contrib/...
- ./instances/kubernetes/...
- ./instances/kubernetes.github.io/...
Kubernetes deliberately decouples service naming/discovery and load balancing from application implementation, since the latter is diverse, open-ended, and may be comprised of multiple deployments, such as in the case of a canary deployment. Similarly, a service / load balancer needs to span deployments across multiple clusters. Therefore, it would be useful to separate resources that should be replicated in such scenarios from those that should not.
**Ideally this would work with all kubectl commands that accept --filename/-f**, not just apply: create, delete, get, replace, etc. We could also create a command that just performed the expansions and output the resulting resources.
```
kubectl apply -f ./instances/test-infra/
```
We need a ["kubectl set serviceaccount" command](https://github.com/kubernetes/kubernetes/pull/48432) to be able to inject an application-specific service account. In general, I think we’ll want to build some environment-management tooling, to create namespaces, service accounts, RBAC rules, policy resources (LimitRange, PodSecurityPolicy, etc.), and so on, perhaps not unlike Openshift’s [new-project flow](https://docs.openshift.org/latest/admin_guide/managing_projects.html#modifying-the-template-for-new-projects).
It would be convenient for tools, UIs, etc. to instantiate a representation of the application, including such information as the application name, labels/selector, service account, package base and version, description, icon, and so on in a CRD. It could also make sense for this resource to serve as the owner of all resources comprising the application, to ensure proper cleanup (except perhaps, optionally, PVCs). Rather than create another representation, the proposed manifest formats could be tweaked to be more amenable to direct instantiation as objects. One possible approach could be to create a Package resource that includes the union of the information in the base manifest and overlay manifest, and then patch the overlay’s Package onto the base’s Package, much like the other resources. While the file paths in such a live resource probably wouldn’t be of much use, it might reduce confusion to rewrite the relative paths of the two subdirectories to be relative to the same root.
Note, again, that this proposal does not include explicit inline parameters ([example of parameterization run amuck](https://github.com/kubernetes/charts/blob/master/stable/artifactory/templates/artifactory-deployment.yaml)). Instead, it relies on forking, overlays, and Kubernetes-specific tooling.