Merge pull request #8 from justinsb/fintegration

cloudup & nodeup: direct, terraform, cloud-init
2016-05-06 16:55:50 -07:00 · 2016-05-06 16:55:50 -07:00 · 72a79d7998
parent be661c34ef d4c2cfaae7
commit 72a79d7998
189 changed files with 10086 additions and 0 deletions
--- a/upup/.gitignore
+++ b/upup/.gitignore
@ -0,0 +1,2 @@
+vendor/
+.build/
--- a/upup/DEVELOP.md
+++ b/upup/DEVELOP.md
@ -0,0 +1,42 @@
+Random scribblings useful for development...
+
+
+## Developing nodeup
+
+ssh ${HOST} sudo mkdir -p /opt/nodeup/state
+ssh ${HOST} sudo chown -R ${USER} /opt/nodeup
+
+go install k8s.io/kube-deploy/upup/... && rsync ~/k8s/bin/nodeup ${HOST}:/opt/nodeup/nodeup && rsync --delete -avz trees/ ${HOST}:/opt/nodeup/trees/ \
+&& rsync state/node.yaml ${HOST}:/opt/nodeup/state/node.yaml \
+&& ssh ${HOST} sudo /opt/nodeup/nodeup --v=2  --template=/opt/nodeup/trees/nodeup --state=/opt/nodeup/state --tags=kubernetes_pool,debian_family,gce,systemd
+
+
+# Random misc
+
+Extract the master node config from a terraform output
+
+cat tf/k8s.tf.json | jq -r '.resource.google_compute_instance["kubernetes-master"].metadata.config' > state/node.yaml
+
+
+
+TODOS
+======
+
+* Implement number-of-tags prioritization
+* Allow files ending in .md to be ignored.  Useful for comments.
+* Better dependency tracking on systemd services?
+* Automatically use different file mode if starts with #! ?
+* Support .static under files to allow for files ending in .template?
+* How to inherit options
+* Allow customization of ordering?  Maybe prefix based.
+* Cache hashes in-process (along with timestamp?) so we don't hash the kubernetes binary bundle repeatedly
+* Fix the fact that we hash assets twice
+* Confirm that we drop support for init.d
+* Can we just use JSON custom marshalling instead of all our reflection stuff (or at least lighten the load)
+
+* Do we officially publish https://storage.googleapis.com/kubernetes-release/release/v1.2.2/kubernetes-server-linux-amd64.tar.gz (ie just the server tar.gz)?
+
+* Need to start docker-healthcheck once
+
+* Can we replace some or all of nodeup config with pkg/apis/componentconfig/types.go ?
+ 
--- a/upup/Makefile
+++ b/upup/Makefile
@ -0,0 +1,31 @@
+gocode:
+	glide install
+	go install k8s.io/kube-deploy/upup/cmd/...
+
+tar: gocode
+	rm -rf .build/tar
+	mkdir -p .build/tar/nodeup/root
+	cp ${GOPATH}/bin/nodeup .build/tar/nodeup/root
+	cp -r models/nodeup/ .build/tar/nodeup/root/model/
+	tar czvf .build/nodeup.tar.gz -C .build/tar/ .
+	tar tvf .build/nodeup.tar.gz
+	(sha1sum .build/nodeup.tar.gz | cut -d' ' -f1) > .build/nodeup.tar.gz.sha1 
+
+upload: tar
+	rm -rf .build/s3
+	mkdir -p .build/s3/nodeup
+	cp .build/nodeup.tar.gz .build/s3/nodeup/
+	cp .build/nodeup.tar.gz.sha1 .build/s3/nodeup/
+	aws s3 sync .build/s3/ s3://kubeupv2/
+	aws s3api put-object-acl --bucket kubeupv2 --key nodeup/nodeup.tar.gz --acl public-read
+	aws s3api put-object-acl --bucket kubeupv2 --key nodeup/nodeup.tar.gz.sha1 --acl public-read
+
+push: tar
+	scp .build/nodeup.tar.gz ${TARGET}:/tmp/
+	ssh ${TARGET} sudo tar zxf /tmp/nodeup.tar.gz -C /var/cache/kubernetes-install
+
+push-dry: push
+	ssh ${TARGET} sudo SKIP_PACKAGE_UPDATE=1 /var/cache/kubernetes-install/nodeup/root/nodeup --conf=metadata://gce/config --dryrun --v=8 --template=/var/cache/kubernetes-install/nodeup/root/model
+
+push-run: push
+	ssh ${TARGET} sudo SKIP_PACKAGE_UPDATE=1 /var/cache/kubernetes-install/nodeup/root/nodeup --conf=metadata://gce/config --v=8 --template=/var/cache/kubernetes-install/nodeup/root/model
--- a/upup/README.md
+++ b/upup/README.md
@ -0,0 +1,51 @@
+## UpUp - CloudUp & NodeUp
+
+CloudUp and NodeUp are two tools that are aiming to replace kube-up:
+the easiest way to get a production Kubernetes up and running.
+
+(Currently work in progress, but working.  Some of these statements are forward-looking.)
+
+Some of the more interesting features:
+
+* Written in go, so hopefully easier to maintain and extend, as complexity inevitably increases
+* Uses a state-sync model, so we get things like a dry-run mode and idempotency automatically
+* Based on a simple meta-model defined in a directory tree
+* Can produce configurations in other formats (currently Terraform & Cloud-Init), so that we can have working
+  configurations for other tools also.
+
+## Bringing up a cluster
+
+Set `YOUR_GCE_PROJECT`, then:
+
+```
+cd upup
+make
+${GOPATH}/bin/cloudup --v=0 --logtostderr -cloud=gce -zone=us-central1-f -project=$YOUR_GCE_PROJECT -name=kubernetes -kubernetes-version=1.2.2
+```
+
+If you have problems, please set `--v=8 --logtostderr` and open an issue, and ping justinsb on slack!
+
+For now, we don't build a local kubectl file.  So just ssh to the master, and run kubectl from there:
+
+```
+gcloud compute ssh kubernetes-master
+...
+kubectl get nodes
+kubectl get pods --all-namespaces
+```
+
+## Other interesting modes:
+
+See changes that would be applied: `${GOPATH}/bin/cloudup --dryrun`
+
+Build a terrform model: `${GOPATH}/bin/cloudup $NORMAL_ARGS --target=terraform > tf/k8s.tf.json`
+
+# How it works
+
+Everything is driven by a local configuration directory tree, called the "model".  The model represents
+the desired state of the world.
+
+Each file in the tree describes a Task.
+
+On the nodeup side, Tasks can manage files, systemd services, packages etc.
+On the cloudup side, Tasks manage cloud resources: instances, networks, disks etc.
--- a/upup/cmd/cloudup/main.go
+++ b/upup/cmd/cloudup/main.go
@ -0,0 +1,244 @@
+package main
+
+import (
+	"flag"
+	"fmt"
+	"github.com/golang/glog"
+	"io/ioutil"
+	"k8s.io/kube-deploy/upup/pkg/fi"
+	"k8s.io/kube-deploy/upup/pkg/fi/cloudup"
+	"k8s.io/kube-deploy/upup/pkg/fi/cloudup/gce"
+	"k8s.io/kube-deploy/upup/pkg/fi/cloudup/gcetasks"
+	"k8s.io/kube-deploy/upup/pkg/fi/cloudup/terraform"
+	"k8s.io/kube-deploy/upup/pkg/fi/loader"
+	"k8s.io/kube-deploy/upup/pkg/fi/utils"
+	"os"
+	"path"
+	"strings"
+)
+
+func main() {
+	dryrun := false
+	flag.BoolVar(&dryrun, "dryrun", false, "Don't create cloud resources; just show what would be done")
+	target := "direct"
+	flag.StringVar(&target, "target", target, "Target - direct, terraform")
+	configFile := ""
+	flag.StringVar(&configFile, "conf", configFile, "Configuration file to load")
+	modelDir := "models/cloudup"
+	flag.StringVar(&modelDir, "model", modelDir, "Source directory to use as model")
+	stateDir := "./state"
+	flag.StringVar(&stateDir, "state", stateDir, "Directory to use to store local state")
+	nodeModelDir := "models/nodeup"
+	flag.StringVar(&nodeModelDir, "nodemodel", nodeModelDir, "Source directory to use as model for node configuration")
+
+	// TODO: Replace all these with a direct binding to the CloudConfig
+	// (we have plenty of reflection helpers if one isn't already available!)
+	config := &cloudup.CloudConfig{}
+	flag.StringVar(&config.CloudProvider, "cloud", config.CloudProvider, "Cloud provider to use - gce, aws")
+	flag.StringVar(&config.Zone, "zone", config.Zone, "Cloud zone to target (warning - will be replaced by region)")
+	flag.StringVar(&config.Project, "project", config.Project, "Project to use (must be set on GCE)")
+	flag.StringVar(&config.ClusterName, "name", config.ClusterName, "Name for cluster")
+	flag.StringVar(&config.KubernetesVersion, "kubernetes-version", config.KubernetesVersion, "Version of kubernetes to run")
+	//flag.StringVar(&config.Region, "region", config.Region, "Cloud region to target")
+
+	flag.Parse()
+
+	if dryrun {
+		target = "dryrun"
+	}
+
+	cmd := &CreateClusterCmd{
+		Config:       config,
+		ModelDir:     modelDir,
+		StateDir:     stateDir,
+		Target:       target,
+		NodeModelDir: nodeModelDir,
+	}
+
+	if configFile != "" {
+		//confFile := path.Join(cmd.StateDir, "kubernetes.yaml")
+		err := cmd.LoadConfig(configFile)
+		if err != nil {
+			glog.Errorf("error loading config: %v", err)
+			os.Exit(1)
+		}
+	}
+
+	err := cmd.Run()
+	if err != nil {
+		glog.Errorf("error running command: %v", err)
+		os.Exit(1)
+	}
+
+	glog.Infof("Completed successfully")
+}
+
+type CreateClusterCmd struct {
+	// Config is the cluster configuration
+	Config *cloudup.CloudConfig
+	// ModelDir is the directory in which the cloudup model is found
+	ModelDir string
+	// StateDir is a directory in which we store state (such as the PKI tree)
+	StateDir string
+	// Target specifies how we are operating e.g. direct to GCE, or AWS, or dry-run, or terraform
+	Target string
+	// The directory in which the node model is found
+	NodeModelDir string
+}
+
+func (c *CreateClusterCmd) LoadConfig(configFile string) error {
+	conf, err := ioutil.ReadFile(configFile)
+	if err != nil {
+		return fmt.Errorf("error loading configuration file %q: %v", configFile, err)
+	}
+	err = utils.YamlUnmarshal(conf, c.Config)
+	if err != nil {
+		return fmt.Errorf("error parsing configuration file %q: %v", configFile, err)
+	}
+	return nil
+}
+
+func (c *CreateClusterCmd) Run() error {
+	if c.StateDir == "" {
+		return fmt.Errorf("state dir is required")
+	}
+
+	if c.Config.CloudProvider == "" {
+		return fmt.Errorf("must specify CloudProvider.  Specify with -cloud")
+	}
+
+	tags := make(map[string]struct{})
+
+	l := &cloudup.Loader{}
+	l.Init()
+
+	caStore, err := fi.NewFilesystemCAStore(path.Join(c.StateDir, "pki"))
+	if err != nil {
+		return fmt.Errorf("error building CA store: %v", err)
+	}
+	secretStore, err := fi.NewFilesystemSecretStore(path.Join(c.StateDir, "secrets"))
+	if err != nil {
+		return fmt.Errorf("error building secret store: %v", err)
+	}
+
+	if len(c.Config.Assets) == 0 {
+		if c.Config.KubernetesVersion == "" {
+			return fmt.Errorf("Must either specify a KubernetesVersion (-kubernetes-version) or provide an asset with the release bundle")
+		}
+		defaultReleaseAsset := fmt.Sprintf("https://storage.googleapis.com/kubernetes-release/release/v%s/kubernetes-server-linux-amd64.tar.gz", c.Config.KubernetesVersion)
+		glog.Infof("Adding default kubernetes release asset: %s", defaultReleaseAsset)
+		// TODO: Verify it exists, get the hash (that will check that KubernetesVersion is valid)
+		c.Config.Assets = append(c.Config.Assets, defaultReleaseAsset)
+	}
+
+	if c.Config.NodeUp.Location == "" {
+		location := "https://kubeupv2.s3.amazonaws.com/nodeup/nodeup.tar.gz"
+		glog.Infof("Using default nodeup location: %q", location)
+		c.Config.NodeUp.Location = location
+	}
+
+	var cloud fi.Cloud
+
+	var project string
+	var region string
+
+	checkExisting := true
+
+	switch c.Config.CloudProvider {
+	case "gce":
+		tags["_gce"] = struct{}{}
+		l.AddTypes(map[string]interface{}{
+			"persistentDisk":       &gcetasks.PersistentDisk{},
+			"instance":             &gcetasks.Instance{},
+			"instanceTemplate":     &gcetasks.InstanceTemplate{},
+			"network":              &gcetasks.Network{},
+			"managedInstanceGroup": &gcetasks.ManagedInstanceGroup{},
+			"firewallRule":         &gcetasks.FirewallRule{},
+			"ipAddress":            &gcetasks.IPAddress{},
+		})
+
+		// For now a zone to be specified...
+		// This will be replace with a region when we go full HA
+		zone := c.Config.Zone
+		if zone == "" {
+			return fmt.Errorf("Must specify a zone (use -zone)")
+		}
+		tokens := strings.Split(zone, "-")
+		if len(tokens) <= 2 {
+			return fmt.Errorf("Invalid Zone: %v", zone)
+		}
+		region = tokens[0] + "-" + tokens[1]
+
+		project = c.Config.Project
+		if project == "" {
+			return fmt.Errorf("project is required for GCE")
+		}
+		gceCloud, err := gce.NewGCECloud(region, project)
+		if err != nil {
+			return err
+		}
+		cloud = gceCloud
+
+	default:
+		return fmt.Errorf("unknown CloudProvider %q", c.Config.CloudProvider)
+	}
+
+	l.Tags = tags
+	l.CAStore = caStore
+	l.SecretStore = secretStore
+	l.StateDir = c.StateDir
+	l.NodeModelDir = c.NodeModelDir
+	l.OptionsLoader = loader.NewOptionsLoader(c.Config)
+
+	taskMap, err := l.Build(c.ModelDir)
+	if err != nil {
+		glog.Exitf("error building: %v", err)
+	}
+
+	if c.Config.ClusterName == "" {
+		return fmt.Errorf("ClusterName is required")
+	}
+
+	if c.Config.Zone == "" {
+		return fmt.Errorf("Zone is required")
+	}
+
+	var target fi.Target
+
+	switch c.Target {
+	case "direct":
+		switch c.Config.CloudProvider {
+		case "gce":
+			target = gce.NewGCEAPITarget(cloud.(*gce.GCECloud))
+		default:
+			return fmt.Errorf("direct configuration not supported with CloudProvider:%q", c.Config.CloudProvider)
+		}
+
+	case "terraform":
+		checkExisting = false
+		target = terraform.NewTerraformTarget(region, project, os.Stdout)
+
+	case "dryrun":
+		target = fi.NewDryRunTarget(os.Stdout)
+	default:
+		return fmt.Errorf("unsupported target type %q", c.Target)
+	}
+
+	context, err := fi.NewContext(target, cloud, caStore, checkExisting)
+	if err != nil {
+		glog.Exitf("error building context: %v", err)
+	}
+	defer context.Close()
+
+	err = context.RunTasks(taskMap)
+	if err != nil {
+		glog.Exitf("error running tasks: %v", err)
+	}
+
+	err = target.Finish(taskMap)
+	if err != nil {
+		glog.Exitf("error closing target: %v", err)
+	}
+
+	return nil
+}
--- a/upup/cmd/nodeup/main.go
+++ b/upup/cmd/nodeup/main.go
@ -0,0 +1,51 @@
+package main
+
+import (
+	"flag"
+	"fmt"
+	"github.com/golang/glog"
+	"k8s.io/kube-deploy/upup/pkg/fi/nodeup"
+	"os"
+)
+
+func main() {
+	flagModel := "model"
+	flag.StringVar(&flagModel, "model", flagModel, "directory to use as model for desired configuration")
+	var flagConf string
+	flag.StringVar(&flagConf, "conf", "node.yaml", "configuration location")
+	var flagAssetDir string
+	flag.StringVar(&flagAssetDir, "assets", "/var/cache/nodeup", "the location for the local asset cache")
+
+	dryrun := false
+	flag.BoolVar(&dryrun, "dryrun", false, "Don't create cloud resources; just show what would be done")
+	target := "direct"
+	flag.StringVar(&target, "target", target, "Target - direct, cloudinit")
+
+	flag.Parse()
+
+	if dryrun {
+		target = "dryrun"
+	}
+
+	flag.Set("logtostderr", "true")
+	flag.Parse()
+
+	if flagConf == "" {
+		glog.Exitf("--conf is required")
+	}
+
+	config := &nodeup.NodeConfig{}
+	cmd := &nodeup.NodeUpCommand{
+		Config:         config,
+		ConfigLocation: flagConf,
+		ModelDir:       flagModel,
+		Target:         target,
+		AssetDir:       flagAssetDir,
+	}
+	err := cmd.Run(os.Stdout)
+	if err != nil {
+		glog.Exitf("error running nodeup: %v", err)
+		os.Exit(1)
+	}
+	fmt.Printf("success")
+}
--- a/upup/glide.lock
+++ b/upup/glide.lock
@ -0,0 +1,53 @@
+hash: 67b60195692c44c9e3be82ced106118324e5de46357a1c11a2942aef8675816e
+updated: 2016-05-06T15:48:38.735466083-04:00
+imports:
+- name: github.com/cloudfoundry-incubator/candiedyaml
+  version: 99c3df83b51532e3615f851d8c2dbb638f5313bf
+- name: github.com/ghodss/yaml
+  version: e8e0db9016175449df0e9c4b6e6995a9433a395c
+- name: github.com/golang/glog
+  version: 23def4e6c14b4da8ac2ed8007337bc5eb5007998
+- name: github.com/golang/protobuf
+  version: 7cc19b78d562895b13596ddce7aafb59dd789318
+  subpackages:
+  - proto
+- name: golang.org/x/net
+  version: 7e42c0e1329bb108f7376a7618a2871ab90f1c4d
+  subpackages:
+  - context
+  - context/ctxhttp
+- name: golang.org/x/oauth2
+  version: e86e2718db89775a4604abc10a5d3a5672e7336e
+  subpackages:
+  - google
+  - internal
+  - jws
+  - jwt
+- name: google.golang.org/api
+  version: f9a4669e07732c84854dce1f5c451c22427228fb
+  subpackages:
+  - compute/v1
+  - googleapi
+  - storage/v1
+  - gensupport
+  - googleapi/internal/uritemplates
+- name: google.golang.org/appengine
+  version: e234e71924d4aa52444bc76f2f831f13fa1eca60
+  subpackages:
+  - urlfetch
+  - internal
+  - internal/app_identity
+  - internal/modules
+  - internal/urlfetch
+  - internal/base
+  - internal/datastore
+  - internal/log
+  - internal/remote_api
+- name: google.golang.org/cloud
+  version: 200292f09e3aaa34878d801ab71fe823b1f7d36a
+  subpackages:
+  - compute/metadata
+  - internal
+- name: google.golang.org/grpc
+  version: 9604a2bb7dd81d87c2873a9580258465f3c311c8
+devImports: []
--- a/upup/glide.yaml
+++ b/upup/glide.yaml
@ -0,0 +1,16 @@
+package: k8s.io/kube-deploy/upup
+import:
+- package: github.com/ghodss/yaml
+- package: github.com/golang/glog
+- package: golang.org/x/net
+  subpackages:
+  - context
+- package: golang.org/x/oauth2
+  subpackages:
+  - google
+- package: google.golang.org/api
+  subpackages:
+  - compute/v1
+  - googleapi
+  - storage/v1
+- package: google.golang.org/grpc
--- a/upup/models/cloudup/_gce/defaults.options
+++ b/upup/models/cloudup/_gce/defaults.options
@ -0,0 +1,53 @@
+ClusterName: {{ .InstancePrefix }}
+InstancePrefix: kubernetes
+AllocateNodeCIDRs: true
+Multizone: true
+
+ServiceClusterIPRange: 10.0.0.0/16
+ClusterIPRange: 10.244.0.0/16
+MasterInternalIP: 172.20.0.9
+MasterIPRange: 10.246.0.0/24
+NetworkProvider: none
+
+AdmissionControl: NamespaceLifecycle,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota,PersistentVolumeLabel
+
+EnableClusterMonitoring: none
+EnableL7LoadBalancing: none
+EnableClusterUI: true
+
+EnableClusterDNS: true
+DNSReplicas: 1
+DNSServerIP: 10.0.0.10
+DNSDomain: cluster.local
+
+EnableClusterLogging: true
+EnableNodeLogging: true
+LoggingDestination: elasticsearch
+ElasticsearchLoggingReplicas: 1
+
+MasterImage: k8s-1-2-debian-jessie-amd64-2016-04-17
+MasterName: {{ .InstancePrefix }}-master
+MasterTag: {{ .InstancePrefix }}-master
+{{ if gt .NodeCount 500 }}
+MasterMachineType: n1-standard-32
+{{ else if gt .NodeCount 250 }}
+MasterMachineType: n1-standard-16
+{{ else if gt .NodeCount 100 }}
+MasterMachineType: n1-standard-8
+{{ else if gt .NodeCount 10 }}
+MasterMachineType: n1-standard-4
+{{ else if gt .NodeCount 5 }}
+MasterMachineType: n1-standard-2
+{{ else }}
+MasterMachineType: n1-standard-1
+{{ end }}
+MasterVolumeType: pd-ssd
+MasterVolumeSize: 20
+
+NodeImage: k8s-1-2-debian-jessie-amd64-2016-04-17
+NodeCount: 2
+NodeTag: {{ .InstancePrefix }}-minion
+NodeInstancePrefix: {{ .InstancePrefix }}-minion
+NodeMachineType: n1-standard-2
+
+KubeUser: admin
--- a/upup/models/cloudup/_gce/master.yaml
+++ b/upup/models/cloudup/_gce/master.yaml
@ -0,0 +1,44 @@
+# TODO: Support multiple masters
+
+persistentDisk/{{ .MasterName }}-pd:
+  zone: {{ .Zone }}
+  sizeGB: {{ or .MasterVolumeSize 20 }}
+  volumeType: {{ or .MasterVolumeType "pd-ssd" }}
+
+# Open master HTTPS
+firewallRule/{{ .MasterName }}-https:
+  network: network/default
+  sourceRanges: 0.0.0.0/0
+  targetTags: {{ .MasterTag }}
+  allowed: tcp:443
+
+# Allocate master IP
+ipAddress/{{ .MasterName }}-ip:
+  address: {{ .MasterPublicIP }}
+
+# Master instance
+instance/{{ .MasterName }}:
+  ipaddress: ipAddress/{{ .MasterName }}-ip
+  zone: {{ .Zone }}
+  machineType: {{ .MasterMachineType }}
+  image: {{ .MasterImage }}
+  tags: {{ .MasterTag }}
+  network: network/default
+  scopes:
+    - storage-ro
+    - compute-rw
+    - monitoring
+    - logging-write
+  canIpForward: true
+  disks:
+    master-pd: persistentDisk/{{ .MasterName }}-pd
+  metadata:
+    #kube-env: resources/kube-env
+{{ if eq .NodeInit "cloudinit" }}
+    config: resources/cloudinit.yaml _kubernetes_master
+{{ else }}
+    startup-script: resources/nodeup.sh
+    config: resources/config.yaml _kubernetes_master
+{{ end }}
+    cluster-name: resources/cluster-name
+  preemptible: false
--- a/upup/models/cloudup/_gce/network.yaml
+++ b/upup/models/cloudup/_gce/network.yaml
@ -0,0 +1,20 @@
+{{ $networkName := "default" }}
+
+network/{{ $networkName }}:
+  cidr: 10.240.0.0/16
+
+# Allow all internal traffic
+firewallRule/{{ $networkName }}-default-internal:
+  network: network/{{$networkName}}
+  sourceRanges: 10.0.0.0/8
+  allowed:
+    - tcp:1-65535
+    - udp:1-65535
+    - icmp
+
+# SSH is open to the world
+firewallRule/{{ $networkName }}-default-ssh:
+  network: network/default
+  sourceRanges: 0.0.0.0/0
+  allowed: tcp:22
+
--- a/upup/models/cloudup/_gce/nodes.yaml
+++ b/upup/models/cloudup/_gce/nodes.yaml
@ -0,0 +1,48 @@
+# TODO: Support multiple instance groups
+
+instanceTemplate/{{ .NodeInstancePrefix }}-template:
+  network: network/default
+  machineType: {{ .NodeMachineType }}
+  # TODO: Make configurable
+  bootDiskType: pd-standard
+  bootDiskSizeGB: 100
+  bootDiskImage: {{ .NodeImage }}
+  canIpForward: true
+  # TODO: Support preemptible nodes?
+  preemptible: false
+  scopes:
+    - compute-rw
+    - monitoring
+    - logging-write
+    - storage-ro
+  metadata:
+    # kube-env: resources/kube-env
+{{ if eq .NodeInit "cloudinit" }}
+    # TODO: we should probably always store the config somewhere
+    config: resources/cloudinit.yaml _kubernetes_master
+{{ else }}
+    startup-script: resources/nodeup.sh
+    config: resources/config.yaml _kubernetes_pool
+{{ end }}
+    cluster-name: resources/cluster-name
+  tags:
+    - {{ .NodeTag }}
+
+managedInstanceGroup/{{ .NodeInstancePrefix }}-group:
+  zone: {{ .Zone }}
+  baseInstanceName: {{ .NodeInstancePrefix }}
+  targetSize: {{ .NodeCount }}
+  instanceTemplate: instanceTemplate/{{ .NodeInstancePrefix }}-template
+
+# Allow traffic from nodes -> nodes
+firewallRule/{{ .NodeTag }}-all:
+  network: network/default
+  sourceRanges: {{ .ClusterIPRange }}
+  targetTags: {{ .NodeTag }}
+  allowed:
+    - tcp
+    - udp
+    - icmp
+    - esp
+    - ah
+    - sctp
--- a/upup/models/cloudup/_gce/resources/cloudinit.yaml.template
+++ b/upup/models/cloudup/_gce/resources/cloudinit.yaml.template
@ -0,0 +1 @@
+{{ BuildNodeConfig "cloudinit" "resources/config.yaml.template" Args }}
--- a/upup/models/cloudup/_gce/resources/cluster-name.template
+++ b/upup/models/cloudup/_gce/resources/cluster-name.template
@ -0,0 +1 @@
+{{ .ClusterName }}
--- a/upup/models/cloudup/_gce/resources/config.yaml.template
+++ b/upup/models/cloudup/_gce/resources/config.yaml.template
@ -0,0 +1,39 @@
+Kubelet:
+  Certificate: {{ Base64Encode (CA.Cert "kubelet").AsString }}
+  Key: {{ Base64Encode (CA.PrivateKey "kubelet").AsString }}
+
+NodeUp:
+  Location: https://kubeupv2.s3.amazonaws.com/nodeup/nodeup.tar.gz
+
+CACertificate: {{ Base64Encode (CA.Cert "ca").AsString }}
+
+APIServer:
+  Certificate: {{ Base64Encode (CA.Cert "master").AsString }}
+  Key: {{ Base64Encode (CA.PrivateKey "master").AsString }}
+
+KubeUser: {{ .KubeUser }}
+KubePassword: {{ (Secrets.Secret "kube").AsString }}
+
+Tokens:
+  admin: {{ (Secrets.Secret "admin").AsString }}
+  kubelet: {{ (Secrets.Secret "kubelet").AsString }}
+  kube-proxy: {{ (Secrets.Secret "kube-proxy").AsString }}
+  "system:scheduler": {{ (Secrets.Secret "system:scheduler").AsString }}
+  "system:controller_manager": {{ (Secrets.Secret "system:controller_manager").AsString }}
+  "system:logging": {{ (Secrets.Secret "system:logging").AsString }}
+  "system:monitoring": {{ (Secrets.Secret "system:monitoring").AsString }}
+  "system:dns": {{ (Secrets.Secret "system:dns").AsString }}
+
+Tags:
+{{ range $tag := Args }}
+  - {{ $tag }}
+{{ end }}
+  - _gce
+  - _jessie
+  - _debian_family
+  - _systemd
+
+Assets:
+{{ range $asset := .Assets }}
+  - {{ $asset }}
+{{ end }}
--- a/upup/models/cloudup/_gce/resources/kube-env.template
+++ b/upup/models/cloudup/_gce/resources/kube-env.template
@ -0,0 +1,150 @@
+INSTANCE_PREFIX: {{ .InstancePrefix }}
+NODE_INSTANCE_PREFIX: {{ .NodeInstancePrefix }}
+CLUSTER_IP_RANGE: {{ .ClusterIPRange }}
+
+#{
+#url, hash, err := k.ServerBinaryTar.Resolve(fi.HashAlgorithmSHA1)
+#if err != nil {
+#return nil, err
+#}
+#SERVER_BINARY_TAR_URL"] = url
+#SERVER_BINARY_TAR_HASH"] = hash
+#}
+
+#{
+#url, hash, err := k.SaltTar.Resolve(fi.HashAlgorithmSHA1)
+#if err != nil {
+#return nil, err
+#}
+#SALT_TAR_URL"] = url
+#SALT_TAR_HASH"] = hash
+#}
+
+SERVICE_CLUSTER_IP_RANGE: {{ .ServiceClusterIPRange }}
+
+KUBERNETES_MASTER_NAME: {{ .MasterName }}
+
+ALLOCATE_NODE_CIDRS: {{ .AllocateNodeCIDRs }}
+
+ENABLE_CLUSTER_MONITORING: {{ .EnableClusterMonitoring }}
+ENABLE_L7_LOADBALANCING: {{ .EnableL7LoadBalancing }}
+ENABLE_CLUSTER_LOGGING: {{ .EnableClusterLogging }}
+ENABLE_CLUSTER_UI: {{ .EnableClusterUI }}
+ENABLE_NODE_LOGGING: {{ .EnableNodeLogging }}
+LOGGING_DESTINATION: {{ .LoggingDestination }}
+ELASTICSEARCH_LOGGING_REPLICAS: {{ .ElasticsearchLoggingReplicas }}
+ENABLE_CLUSTER_DNS: {{ .EnableClusterDNS }}
+ENABLE_CLUSTER_REGISTRY: {{ .EnableClusterRegistry }}
+CLUSTER_REGISTRY_DISK: {{ .ClusterRegistryDisk }}
+CLUSTER_REGISTRY_DISK_SIZE: {{ .ClusterRegistryDiskSize }}
+DNS_REPLICAS: {{.DNSReplicas }}
+DNS_SERVER_IP: {{ .DNSServerIP }}
+DNS_DOMAIN: {{ .DNSDomain }}
+
+KUBELET_TOKEN: {{ .KubeletToken }}
+KUBE_PROXY_TOKEN: {{ .KubeProxyToken }}
+ADMISSION_CONTROL: {{ .AdmissionControl }}
+MASTER_IP_RANGE: {{ .MasterIPRange }}
+RUNTIME_CONFIG: {{ .RuntimeConfig }}
+
+CA_CERT: {{ Base64Encode (CA.Cert "ca").AsString }}
+KUBELET_CERT: {{ Base64Encode (CA.Cert "kubelet").AsString }}
+KUBELET_KEY: {{ Base64Encode (CA.PrivateKey "kubelet").AsString }}
+
+NETWORK_PROVIDER: {{ .NetworkProvider }}
+HAIRPIN_MODE: {{ .HairpinMode }}
+OPENCONTRAIL_TAG: {{ .OpencontrailTag }}
+OPENCONTRAIL_KUBERNETES_TAG: {{ .OpencontrailKubernetesTag }}
+OPENCONTRAIL_PUBLIC_SUBNET: {{ .OpencontrailPublicSubnet }}
+E2E_STORAGE_TEST_ENVIRONMENT: {{ .E2EStorageTestEnvironment }}
+KUBE_IMAGE_TAG: {{ .KubeImageTag }}
+KUBE_DOCKER_REGISTRY: {{ .KubeDockerRegistry }}
+KUBE_ADDON_REGISTRY: {{ .KubeAddonRegistry }}
+MULTIZONE: {{ .Multizone }}
+NON_MASQUERADE_CIDR: {{ .NonMasqueradeCidr }}
+
+KUBELET_PORT: {{ .KubeletPort }}
+
+KUBE_APISERVER_REQUEST_TIMEOUT: {{ .KubeApiserverRequestTimeout }}
+
+TERMINATED_POD_GC_THRESHOLD: {{ .TerminatedPodGcThreshold }}
+
+#if k.OsDistribution == "trusty" {
+#KUBE_MANIFESTS_TAR_URL: .KubeManifestsTarURL }}
+#KUBE_MANIFESTS_TAR_HASH: .KubeManifestsTarSha256 }}
+#}
+
+TEST_CLUSTER: {{ .TestCluster }}
+
+KUBELET_TEST_ARGS: {{ .KubeletTestArgs }}
+
+KUBELET_TEST_LOG_LEVEL: {{ .KubeletTestLogLevel }}
+
+DOCKER_TEST_LOG_LEVEL: {{ .DockerTestLogLevel }}
+
+ENABLE_CUSTOM_METRICS: {{ .EnableCustomMetrics }}
+
+# if .Target.IsMaster
+
+# If the user requested that the master be part of the cluster, set the
+# environment variable to program the master kubelet to register itself.
+{{ if .RegisterMasterKubelet }}
+KUBELET_APISERVER: {{ .MasterName }}
+{{ end }}
+
+KUBERNETES_MASTER: true
+KUBE_USER: {{ .KubeUser }}
+KUBE_PASSWORD: {{ .KubePassword }}
+KUBE_BEARER_TOKEN: {{ .BearerToken }}
+MASTER_CERT: {{ Base64Encode (CA.Cert "master").AsString }}
+MASTER_KEY: {{ Base64Encode (CA.PrivateKey "master").AsString }}
+KUBECFG_CERT: {{ Base64Encode (CA.Cert "kubecfg").AsString }}
+KUBECFG_KEY: {{ Base64Encode (CA.PrivateKey "kubecfg").AsString }}
+
+ENABLE_MANIFEST_URL: {{ .EnableManifestURL }}
+MANIFEST_URL: {{ .ManifestURL }}
+MANIFEST_URL_HEADER: {{ .ManifestURLHeader }}
+NUM_NODES: {{.NodeCount }}
+
+APISERVER_TEST_ARGS: {{ .ApiserverTestArgs }}
+
+APISERVER_TEST_LOG_LEVEL: {{ .ApiserverTestLogLevel }}
+
+CONTROLLER_MANAGER_TEST_ARGS: {{ .ControllerManagerTestArgs }}
+
+CONTROLLER_MANAGER_TEST_LOG_LEVEL: {{ .ControllerManagerTestLogLevel }}
+
+SCHEDULER_TEST_ARGS: {{ .SchedulerTestArgs }}
+
+SCHEDULER_TEST_LOG_LEVEL: {{ .SchedulerTestLogLevel }}
+
+# else
+
+# Node-only vars
+
+KUBERNETES_MASTER: false
+ZONE: {{ .Zone }}
+EXTRA_DOCKER_OPTS: {{ .ExtraDockerOpts }}
+MANIFEST_URL: {{ .ManifestURL }}
+
+KUBEPROXY_TEST_ARGS: {{ .KubeProxyTestArgs }}
+
+KUBEPROXY_TEST_LOG_LEVEL: {{ .KubeProxyTestLogLevel }}
+
+# end
+
+NODE_LABELS: {{ .NodeLabels }}
+
+#if k.OsDistribution == "coreos" {
+#// CoreOS-only env vars. TODO(yifan): Make them available on other distros.
+#KUBE_MANIFESTS_TAR_URL:  .KubeManifestsTarURL }}
+#KUBE_MANIFESTS_TAR_HASH:  .KubeManifestsTarSha256 }}
+#KUBERNETES_CONTAINER_RUNTIME:  .ContainerRuntime }}
+#RKT_VERSION: .RktVersion }}
+#RKT_PATH:  .RktPath }}
+#KUBERNETES_CONFIGURE_CBR0:  .KubernetesConfigureCbr0 }}
+#}
+
+# This next bit for changes vs kube-up:
+# https://github.com/kubernetes/kubernetes/issues/23264
+CA_KEY: {{ Base64Encode (CA.PrivateKey "ca").AsString }}
--- a/upup/models/cloudup/_gce/resources/nodeup.sh.template
+++ b/upup/models/cloudup/_gce/resources/nodeup.sh.template
@ -0,0 +1,139 @@
+#!/bin/bash
+# Copyright 2016 The Kubernetes Authors All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -o errexit
+set -o nounset
+set -o pipefail
+
+NODEUP_TAR_URL={{ .NodeUp.Location }}
+NODEUP_TAR_HASH={{ .NodeUp.Hash }}
+
+function ensure-basic-networking() {
+  # Deal with GCE networking bring-up race. (We rely on DNS for a lot,
+  # and it's just not worth doing a whole lot of startup work if this
+  # isn't ready yet.)
+  until getent hosts metadata.google.internal &>/dev/null; do
+    echo 'Waiting for functional DNS (trying to resolve metadata.google.internal)...'
+    sleep 3
+  done
+  until getent hosts $(hostname -f || echo _error_) &>/dev/null; do
+    echo 'Waiting for functional DNS (trying to resolve my own FQDN)...'
+    sleep 3
+  done
+  until getent hosts $(hostname -i || echo _error_) &>/dev/null; do
+    echo 'Waiting for functional DNS (trying to resolve my own IP)...'
+    sleep 3
+  done
+
+  echo "Networking functional on $(hostname) ($(hostname -i))"
+}
+
+function ensure-install-dir() {
+  INSTALL_DIR="/var/cache/kubernetes-install"
+  mkdir -p ${INSTALL_DIR}
+  cd ${INSTALL_DIR}
+}
+
+function curl-metadata() {
+  curl --fail --retry 5 --silent -H 'Metadata-Flavor: Google' "http://metadata/computeMetadata/v1/instance/attributes/${1}"
+}
+
+# Retry a download until we get it. Takes a hash and a set of URLs.
+#
+# $1 is the sha1 of the URL. Can be "" if the sha1 is unknown.
+# $2+ are the URLs to download.
+download-or-bust() {
+  local -r hash="$1"
+  shift 1
+
+  urls=( $* )
+  while true; do
+    for url in "${urls[@]}"; do
+      local file="${url##*/}"
+      rm -f "${file}"
+      if ! curl -f --ipv4 -Lo "${file}" --connect-timeout 20 --retry 6 --retry-delay 10 "${url}"; then
+        echo "== Failed to download ${url}. Retrying. =="
+      elif [[ -n "${hash}" ]] && ! validate-hash "${file}" "${hash}"; then
+        echo "== Hash validation of ${url} failed. Retrying. =="
+      else
+        if [[ -n "${hash}" ]]; then
+          echo "== Downloaded ${url} (SHA1 = ${hash}) =="
+        else
+          echo "== Downloaded ${url} =="
+        fi
+        return
+      fi
+    done
+  done
+}
+
+validate-hash() {
+  local -r file="$1"
+  local -r expected="$2"
+  local actual
+
+  actual=$(sha1sum ${file} | awk '{ print $1 }') || true
+  if [[ "${actual}" != "${expected}" ]]; then
+    echo "== ${file} corrupted, sha1 ${actual} doesn't match expected ${expected} =="
+    return 1
+  fi
+}
+
+function split-commas() {
+  echo $1 | tr "," "\n"
+}
+
+function try-download-release() {
+  # TODO(zmerlynn): Now we REALLY have no excuse not to do the reboot
+  # optimization.
+
+  local -r nodeup_tar_urls=( $(split-commas "${NODEUP_TAR_URL}") )
+  local -r nodeup_tar="${nodeup_tar_urls[0]##*/}"
+  if [[ -n "${NODEUP_TAR_HASH:-}" ]]; then
+    local -r nodeup_tar_hash="${NODEUP_TAR_HASH}"
+  else
+  # TODO: Remove?
+    echo "Downloading binary release sha1 (not found in env)"
+    download-or-bust "" "${nodeup_tar_urls[@]/.tar.gz/.tar.gz.sha1}"
+    local -r nodeup_tar_hash=$(cat "${nodeup_tar}.sha1")
+  fi
+
+  echo "Downloading binary release tar (${nodeup_tar_urls[@]})"
+  download-or-bust "${nodeup_tar_hash}" "${nodeup_tar_urls[@]}"
+
+  echo "Unpacking and checking integrity of nodeup"
+  rm -rf nodeupcurl https://kubeupv2.s3.amazonaws.com/nodeup/nodeup.tar.gz
+
+  tar xzf "${nodeup_tar}" && tar tzf "${nodeup_tar}" > /dev/null
+}
+
+function download-release() {
+  # In case of failure checking integrity of release, retry.
+  until try-download-release; do
+    sleep 15
+    echo "Couldn't download release. Retrying..."
+  done
+
+  echo "Running release install script"
+  ( cd nodeup/root; ./nodeup --conf=metadata://{{ .CloudProvider }}/config --v=8 )
+}
+
+####################################################################################
+
+echo "== nodeup node config starting =="
+ensure-basic-networking
+ensure-install-dir
+download-release
+echo "== nodeup node config done =="
--- a/upup/models/cloudup/pki/kubecfg
+++ b/upup/models/cloudup/pki/kubecfg
@ -0,0 +1,3 @@
+subject:
+  CommonName: kubecfg
+type: client
--- a/upup/models/cloudup/pki/kubelet
+++ b/upup/models/cloudup/pki/kubelet
@ -0,0 +1,3 @@
+subject:
+  CommonName: kubelet
+type: client
--- a/upup/models/cloudup/pki/master
+++ b/upup/models/cloudup/pki/master
@ -0,0 +1,12 @@
+subject:
+  CommonName: kubernetes-master
+type: server
+alternateNames:
+  - kubernetes
+  - kubernetes.default
+  - kubernetes.default.svc
+  - kubernetes.default.svc.{{ .DNSDomain }}
+  - {{ .MasterName }}
+  - {{ .MasterPublicIP }}
+  - {{ .MasterInternalIP }}
+  - {{ .WellKnownServiceIP 1 }}
--- a/upup/models/nodeup/_kubernetes_master/_kube-master-addons/files/etc/kubernetes/kube-master-addons.sh
+++ b/upup/models/nodeup/_kubernetes_master/_kube-master-addons/files/etc/kubernetes/kube-master-addons.sh
@ -0,0 +1,66 @@
+#!/bin/bash
+
+# Copyright 2014 The Kubernetes Authors All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# loadedImageFlags is a bit-flag to track which docker images loaded successfully.
+let loadedImageFlags=0
+
+while true; do
+  restart_docker=false
+
+  if which docker 1>/dev/null 2>&1; then
+
+    timeout 30 docker load -i /srv/salt/kube-bins/kube-apiserver.tar 1>/dev/null 2>&1
+    rc=$?
+    if [[ $rc == 0 ]]; then
+      let loadedImageFlags="$loadedImageFlags|1"
+    elif [[ $rc == 124 ]]; then
+      restart_docker=true
+    fi
+
+    timeout 30 docker load -i /srv/salt/kube-bins/kube-scheduler.tar 1>/dev/null 2>&1
+    rc=$?
+    if [[ $rc == 0 ]]; then
+      let loadedImageFlags="$loadedImageFlags|2"
+    elif [[ $rc == 124 ]]; then
+      restart_docker=true
+    fi
+
+    timeout 30 docker load -i /srv/salt/kube-bins/kube-controller-manager.tar 1>/dev/null 2>&1
+    rc=$?
+    if [[ $rc == 0 ]]; then
+      let loadedImageFlags="$loadedImageFlags|4"
+    elif [[ $rc == 124 ]]; then
+      restart_docker=true
+    fi
+  fi
+
+  # required docker images got installed. exit while loop.
+  if [[ $loadedImageFlags == 7 ]]; then break; fi
+
+  # Sometimes docker load hang, restart docker daemon resolve the issue
+  if [[ $restart_docker ]]; then
+    if ! service docker restart; then # Try systemctl if there's no service command.
+      systemctl restart docker
+    fi
+  fi
+
+  # sleep for 15 seconds before attempting to load docker images again
+  sleep 15
+
+done
+
+# Now exit. After kube-push, salt will notice that the service is down and it
+# will start it and new docker images will be loaded.
--- a/upup/models/nodeup/_kubernetes_master/_kube-master-addons/files/etc/kubernetes/kube-master-addons.sh.meta
+++ b/upup/models/nodeup/_kubernetes_master/_kube-master-addons/files/etc/kubernetes/kube-master-addons.sh.meta
@ -0,0 +1,3 @@
+{
+  "mode": "0755"
+}
--- a/upup/models/nodeup/_kubernetes_master/_kube-master-addons/services/kube-master-addons.service
+++ b/upup/models/nodeup/_kubernetes_master/_kube-master-addons/services/kube-master-addons.service
@ -0,0 +1,9 @@
+[Unit]
+Description=Kubernetes-Master Addon Object Manager
+Documentation=https://github.com/kubernetes/kubernetes
+
+[Service]
+ExecStart=/etc/kubernetes/kube-master-addons.sh
+
+[Install]
+WantedBy=multi-user.target
--- a/upup/models/nodeup/_kubernetes_master/certs/files/srv/kubernetes/ca.crt.template
+++ b/upup/models/nodeup/_kubernetes_master/certs/files/srv/kubernetes/ca.crt.template
@ -0,0 +1 @@
+{{ .CACertificate.AsString }}
--- a/upup/models/nodeup/_kubernetes_master/certs/files/srv/kubernetes/server.cert.template
+++ b/upup/models/nodeup/_kubernetes_master/certs/files/srv/kubernetes/server.cert.template
@ -0,0 +1 @@
+{{ .APIServer.Certificate.AsString }}
--- a/upup/models/nodeup/_kubernetes_master/certs/files/srv/kubernetes/server.key.template
+++ b/upup/models/nodeup/_kubernetes_master/certs/files/srv/kubernetes/server.key.template
@ -0,0 +1 @@
+{{ .APIServer.Key.AsString }}
--- a/upup/models/nodeup/_kubernetes_master/etcd/files/etc/kubernetes/manifests/etcd-events.manifest
+++ b/upup/models/nodeup/_kubernetes_master/etcd/files/etc/kubernetes/manifests/etcd-events.manifest
@ -0,0 +1,65 @@
+{
+"apiVersion": "v1",
+"kind": "Pod",
+"metadata": {
+  "name":"etcd-server-events",
+  "namespace": "kube-system"
+},
+"spec":{
+"hostNetwork": true,
+"containers":[
+    {
+    "name": "etcd-container",
+    "image": "gcr.io/google_containers/etcd:2.2.1",
+    "resources": {
+      "requests": {
+        "cpu": "100m"
+      }
+    },
+    "command": [
+              "/bin/sh",
+              "-c",
+              "/usr/local/bin/etcd --listen-peer-urls http://127.0.0.1:2381 --addr 127.0.0.1:4002 --bind-addr 127.0.0.1:4002 --data-dir /var/etcd/data-events 1>>/var/log/etcd-events.log 2>&1"
+            ],
+    "livenessProbe": {
+      "httpGet": {
+        "host": "127.0.0.1",
+        "port": 4002,
+        "path": "/health"
+      },
+      "initialDelaySeconds": 15,
+      "timeoutSeconds": 15
+    },
+    "ports":[
+      { "name": "serverport",
+        "containerPort": 2381,
+        "hostPort": 2381
+      },{
+       "name": "clientport",
+        "containerPort": 4002,
+        "hostPort": 4002
+      }
+        ],
+    "volumeMounts": [
+      {"name": "varetcd",
+       "mountPath": "/var/etcd",
+       "readOnly": false
+      },
+      {"name": "varlogetcd",
+       "mountPath": "/var/log/etcd-events.log",
+       "readOnly": false
+      }
+     ]
+    }
+],
+"volumes":[
+  { "name": "varetcd",
+    "hostPath": {
+        "path": "/mnt/master-pd/var/etcd"}
+  },
+  { "name": "varlogetcd",
+    "hostPath": {
+        "path": "/var/log/etcd-events.log"}
+  }
+]
+}}
--- a/upup/models/nodeup/_kubernetes_master/etcd/files/etc/kubernetes/manifests/etcd.manifest
+++ b/upup/models/nodeup/_kubernetes_master/etcd/files/etc/kubernetes/manifests/etcd.manifest
@ -0,0 +1,65 @@
+{
+"apiVersion": "v1",
+"kind": "Pod",
+"metadata": {
+  "name":"etcd-server",
+  "namespace": "kube-system"
+},
+"spec":{
+"hostNetwork": true,
+"containers":[
+    {
+    "name": "etcd-container",
+    "image": "gcr.io/google_containers/etcd:2.2.1",
+    "resources": {
+      "requests": {
+        "cpu": "200m"
+      }
+    },
+    "command": [
+              "/bin/sh",
+              "-c",
+              "/usr/local/bin/etcd --listen-peer-urls http://127.0.0.1:2380 --addr 127.0.0.1:4001 --bind-addr 127.0.0.1:4001 --data-dir /var/etcd/data 1>>/var/log/etcd.log 2>&1"
+            ],
+    "livenessProbe": {
+      "httpGet": {
+        "host": "127.0.0.1",
+        "port": 4001,
+        "path": "/health"
+      },
+      "initialDelaySeconds": 15,
+      "timeoutSeconds": 15
+    },
+    "ports":[
+      { "name": "serverport",
+        "containerPort": 2380,
+        "hostPort": 2380
+      },{
+       "name": "clientport",
+        "containerPort": 4001,
+        "hostPort": 4001
+      }
+        ],
+    "volumeMounts": [
+      {"name": "varetcd",
+       "mountPath": "/var/etcd",
+       "readOnly": false
+      },
+      {"name": "varlogetcd",
+       "mountPath": "/var/log/etcd.log",
+       "readOnly": false
+      }
+     ]
+    }
+],
+"volumes":[
+  { "name": "varetcd",
+    "hostPath": {
+        "path": "/mnt/master-pd/var/etcd"}
+  },
+  { "name": "varlogetcd",
+    "hostPath": {
+        "path": "/var/log/etcd.log"}
+  }
+]
+}}
--- a/upup/models/nodeup/_kubernetes_master/etcd/files/var/log/etcd-events.log
+++ b/upup/models/nodeup/_kubernetes_master/etcd/files/var/log/etcd-events.log
--- a/upup/models/nodeup/_kubernetes_master/etcd/files/var/log/etcd-events.log.meta
+++ b/upup/models/nodeup/_kubernetes_master/etcd/files/var/log/etcd-events.log.meta
@ -0,0 +1,3 @@
+{
+  "ifNotExists": true
+}
--- a/upup/models/nodeup/_kubernetes_master/etcd/files/var/log/etcd.log
+++ b/upup/models/nodeup/_kubernetes_master/etcd/files/var/log/etcd.log
--- a/upup/models/nodeup/_kubernetes_master/etcd/files/var/log/etcd.log.meta
+++ b/upup/models/nodeup/_kubernetes_master/etcd/files/var/log/etcd.log.meta
@ -0,0 +1,3 @@
+{
+  "ifNotExists": true
+}
--- a/upup/models/nodeup/_kubernetes_master/kube-addons/files/etc/kubernetes/addons/namespace.yaml
+++ b/upup/models/nodeup/_kubernetes_master/kube-addons/files/etc/kubernetes/addons/namespace.yaml
@ -0,0 +1,4 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: kube-system
--- a/upup/models/nodeup/_kubernetes_master/kube-addons/files/etc/kubernetes/kube-addon-update.sh
+++ b/upup/models/nodeup/_kubernetes_master/kube-addons/files/etc/kubernetes/kube-addon-update.sh
@ -0,0 +1,514 @@
+#!/bin/bash
+
+# Copyright 2015 The Kubernetes Authors All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# The business logic for whether a given object should be created
+# was already enforced by salt, and /etc/kubernetes/addons is the
+# managed result is of that. Start everything below that directory.
+
+# Parameters
+# $1 path to add-ons
+
+
+# LIMITATIONS
+# 1. controllers are not updated unless their name is changed
+# 3. Services will not be updated unless their name is changed,
+#    but for services we actually want updates without name change.
+# 4. Json files are not handled at all. Currently addons must be
+#    in yaml files
+# 5. exit code is probably not always correct (I haven't checked
+#    carefully if it works in 100% cases)
+# 6. There are no unittests
+# 8. Will not work if the total length of paths to addons is greater than
+#    bash can handle. Probably it is not a problem: ARG_MAX=2097152 on GCE.
+# 9. Performance issue: yaml files are read many times in a single execution.
+
+# cosmetic improvements to be done
+# 1. improve the log function; add timestamp, file name, etc.
+# 2. logging doesn't work from files that print things out.
+# 3. kubectl prints the output to stderr (the output should be captured and then
+#    logged)
+
+
+
+# global config
+KUBECTL=${TEST_KUBECTL:-}   # substitute for tests
+KUBECTL=${KUBECTL:-${KUBECTL_BIN:-}}
+KUBECTL=${KUBECTL:-/usr/local/bin/kubectl}
+if [[ ! -x ${KUBECTL} ]]; then
+    echo "ERROR: kubectl command (${KUBECTL}) not found or is not executable" 1>&2
+    exit 1
+fi
+
+# If an add-on definition is incorrect, or a definition has just disappeared
+# from the local directory, the script will still keep on retrying.
+# The script does not end until all retries are done, so
+# one invalid manifest may block updates of other add-ons.
+# Be careful how you set these parameters
+NUM_TRIES=1    # will be updated based on input parameters
+DELAY_AFTER_ERROR_SEC=${TEST_DELAY_AFTER_ERROR_SEC:=10}
+
+
+# remember that you can't log from functions that print some output (because
+# logs are also printed on stdout)
+# $1 level
+# $2 message
+function log() {
+  # manage log levels manually here
+
+  # add the timestamp if you find it useful
+  case $1 in
+    DB3 )
+#        echo "$1: $2"
+        ;;
+    DB2 )
+#        echo "$1: $2"
+        ;;
+    DBG )
+#        echo "$1: $2"
+        ;;
+    INFO )
+        echo "$1: $2"
+        ;;
+    WRN )
+        echo "$1: $2"
+        ;;
+    ERR )
+        echo "$1: $2"
+        ;;
+    * )
+        echo "INVALID_LOG_LEVEL $1: $2"
+        ;;
+  esac
+}
+
+#$1 yaml file path
+function get-object-kind-from-file() {
+    # prints to stdout, so log cannot be used
+    #WARNING: only yaml is supported
+    cat $1 | ${PYTHON} -c '''
+try:
+        import pipes,sys,yaml
+        y = yaml.load(sys.stdin)
+        labels = y["metadata"]["labels"]
+        if ("kubernetes.io/cluster-service", "true") not in labels.iteritems():
+            # all add-ons must have the label "kubernetes.io/cluster-service".
+            # Otherwise we are ignoring them (the update will not work anyway)
+            print "ERROR"
+        else:
+            print y["kind"]
+except Exception, ex:
+        print "ERROR"
+    '''
+}
+
+# $1 yaml file path
+# returns a string of the form <namespace>/<name> (we call it nsnames)
+function get-object-nsname-from-file() {
+    # prints to stdout, so log cannot be used
+    #WARNING: only yaml is supported
+    #addons that do not specify a namespace are assumed to be in "default".
+    cat $1 | ${PYTHON} -c '''
+try:
+        import pipes,sys,yaml
+        y = yaml.load(sys.stdin)
+        labels = y["metadata"]["labels"]
+        if ("kubernetes.io/cluster-service", "true") not in labels.iteritems():
+            # all add-ons must have the label "kubernetes.io/cluster-service".
+            # Otherwise we are ignoring them (the update will not work anyway)
+            print "ERROR"
+        else:
+            try:
+                print "%s/%s" % (y["metadata"]["namespace"], y["metadata"]["name"])
+            except Exception, ex:
+                print "default/%s" % y["metadata"]["name"]
+except Exception, ex:
+        print "ERROR"
+    '''
+}
+
+# $1 addon directory path
+# $2 addon type (e.g. ReplicationController)
+# echoes the string with paths to files containing addon for the given type
+# works only for yaml files (!) (ignores json files)
+function get-addon-paths-from-disk() {
+    # prints to stdout, so log cannot be used
+    local -r addon_dir=$1
+    local -r obj_type=$2
+    local kind
+    local file_path
+    for file_path in $(find ${addon_dir} -name \*.yaml); do
+        kind=$(get-object-kind-from-file ${file_path})
+        # WARNING: assumption that the topmost indentation is zero (I'm not sure yaml allows for topmost indentation)
+        if [[ "${kind}" == "${obj_type}" ]]; then
+            echo ${file_path}
+        fi
+    done
+}
+
+# waits for all subprocesses
+# returns 0 if all of them were successful and 1 otherwise
+function wait-for-jobs() {
+    local rv=0
+    local pid
+    for pid in $(jobs -p); do
+        wait ${pid}
+        if [[ $? -ne 0 ]]; then
+            rv=1;
+            log ERR "error in pid ${pid}"
+        fi
+        log DB2 "pid ${pid} completed, current error code: ${rv}"
+    done
+    return ${rv}
+}
+
+
+function run-until-success() {
+    local -r command=$1
+    local tries=$2
+    local -r delay=$3
+    local -r command_name=$1
+    while [ ${tries} -gt 0 ]; do
+        log DBG "executing: '$command'"
+        # let's give the command as an argument to bash -c, so that we can use
+        # && and || inside the command itself
+        /bin/bash -c "${command}" && \
+            log DB3 "== Successfully executed ${command_name} at $(date -Is) ==" && \
+            return 0
+        let tries=tries-1
+        log INFO "== Failed to execute ${command_name} at $(date -Is). ${tries} tries remaining. =="
+        sleep ${delay}
+    done
+    return 1
+}
+
+# $1 object type
+# returns a list of <namespace>/<name> pairs (nsnames)
+function get-addon-nsnames-from-server() {
+    local -r obj_type=$1
+    "${KUBECTL}" get "${obj_type}" --all-namespaces -o go-template="{{range.items}}{{.metadata.namespace}}/{{.metadata.name}} {{end}}" --api-version=v1 -l kubernetes.io/cluster-service=true
+}
+
+# returns the characters after the last separator (including)
+# If the separator is empty or if it doesn't appear in the string,
+# an empty string is printed
+# $1 input string
+# $2 separator (must be single character, or empty)
+function get-suffix() {
+    # prints to stdout, so log cannot be used
+    local -r input_string=$1
+    local -r separator=$2
+    local suffix
+
+    if [[ "${separator}" == "" ]]; then
+        echo ""
+        return
+    fi
+
+    if  [[ "${input_string}" == *"${separator}"* ]]; then
+        suffix=$(echo "${input_string}" | rev | cut -d "${separator}" -f1 | rev)
+        echo "${separator}${suffix}"
+    else
+        echo ""
+    fi
+}
+
+# returns the characters up to the last '-' (without it)
+# $1 input string
+# $2 separator
+function get-basename() {
+    # prints to stdout, so log cannot be used
+    local -r input_string=$1
+    local -r separator=$2
+    local suffix
+    suffix="$(get-suffix ${input_string} ${separator})"
+    # this will strip the suffix (if matches)
+    echo ${input_string%$suffix}
+}
+
+function delete-object() {
+    local -r obj_type=$1
+    local -r namespace=$2
+    local -r obj_name=$3
+    log INFO "Deleting ${obj_type} ${namespace}/${obj_name}"
+
+    run-until-success "${KUBECTL} delete --namespace=${namespace} ${obj_type} ${obj_name}" ${NUM_TRIES} ${DELAY_AFTER_ERROR_SEC}
+}
+
+function create-object() {
+    local -r obj_type=$1
+    local -r file_path=$2
+
+    local nsname_from_file
+    nsname_from_file=$(get-object-nsname-from-file ${file_path})
+    if [[ "${nsname_from_file}" == "ERROR" ]]; then
+       log INFO "Cannot read object name from ${file_path}. Ignoring"
+       return 1
+    fi
+    IFS='/' read namespace obj_name <<< "${nsname_from_file}"
+
+    log INFO "Creating new ${obj_type} from file ${file_path} in namespace ${namespace}, name: ${obj_name}"
+    # this will keep on failing if the ${file_path} disappeared in the meantime.
+    # Do not use too many retries.
+    run-until-success "${KUBECTL} create --namespace=${namespace} -f ${file_path}" ${NUM_TRIES} ${DELAY_AFTER_ERROR_SEC}
+}
+
+function update-object() {
+    local -r obj_type=$1
+    local -r namespace=$2
+    local -r obj_name=$3
+    local -r file_path=$4
+    log INFO "updating the ${obj_type} ${namespace}/${obj_name} with the new definition ${file_path}"
+    delete-object ${obj_type} ${namespace} ${obj_name}
+    create-object ${obj_type} ${file_path}
+}
+
+# deletes the objects from the server
+# $1 object type
+# $2 a list of object nsnames
+function delete-objects() {
+    local -r obj_type=$1
+    local -r obj_nsnames=$2
+    local namespace
+    local obj_name
+    for nsname in ${obj_nsnames}; do
+        IFS='/' read namespace obj_name <<< "${nsname}"
+        delete-object ${obj_type} ${namespace} ${obj_name} &
+    done
+}
+
+# creates objects from the given files
+# $1 object type
+# $2 a list of paths to definition files
+function create-objects() {
+    local -r obj_type=$1
+    local -r file_paths=$2
+    local file_path
+    for file_path in ${file_paths}; do
+        # Remember that the file may have disappear by now
+        # But we don't want to check it here because
+        # such race condition may always happen after
+        # we check it. Let's have the race
+        # condition happen a bit more often so that
+        # we see that our tests pass anyway.
+        create-object ${obj_type} ${file_path} &
+    done
+}
+
+# updates objects
+# $1 object type
+# $2 a list of update specifications
+# each update specification is a ';' separated pair: <nsname>;<file path>
+function update-objects() {
+    local -r obj_type=$1      # ignored
+    local -r update_spec=$2
+    local objdesc
+    local nsname
+    local obj_name
+    local namespace
+
+    for objdesc in ${update_spec}; do
+        IFS=';' read nsname file_path <<< "${objdesc}"
+        IFS='/' read namespace obj_name <<< "${nsname}"
+
+        update-object ${obj_type} ${namespace} ${obj_name} ${file_path} &
+    done
+}
+
+# Global variables set by function match-objects.
+nsnames_for_delete=""   # a list of object nsnames to be deleted
+for_update=""           # a list of pairs <nsname>;<filePath> for objects that should be updated
+nsnames_for_ignore=""   # a list of object nsnames that will be ignored
+new_files=""            # a list of file paths that weren't matched by any existing objects (these objects must be created now)
+
+
+# $1 path to files with objects
+# $2 object type in the API (ReplicationController or Service)
+# $3 name separator (single character or empty)
+function match-objects() {
+    local -r addon_dir=$1
+    local -r obj_type=$2
+    local -r separator=$3
+
+    # output variables (globals)
+    nsnames_for_delete=""
+    for_update=""
+    nsnames_for_ignore=""
+    new_files=""
+
+    addon_nsnames_on_server=$(get-addon-nsnames-from-server "${obj_type}")
+    # if the api server is unavailable then abandon the update for this cycle 
+    if [[ $? -ne 0 ]]; then
+        log ERR "unable to query ${obj_type} - exiting"
+        exit 1
+    fi
+
+    addon_paths_in_files=$(get-addon-paths-from-disk "${addon_dir}" "${obj_type}")
+
+    log DB2 "addon_nsnames_on_server=${addon_nsnames_on_server}"
+    log DB2 "addon_paths_in_files=${addon_paths_in_files}"
+
+    local matched_files=""
+
+    local basensname_on_server=""
+    local nsname_on_server=""
+    local suffix_on_server=""
+    local nsname_from_file=""
+    local suffix_from_file=""
+    local found=0
+    local addon_path=""
+
+    # objects that were moved between namespaces will have different nsname
+    # because the namespace is included. So they will be treated
+    # like different objects and not updated but deleted and created again
+    # (in the current version update is also delete+create, so it does not matter)
+    for nsname_on_server in ${addon_nsnames_on_server}; do
+        basensname_on_server=$(get-basename ${nsname_on_server} ${separator})
+        suffix_on_server="$(get-suffix ${nsname_on_server} ${separator})"
+
+        log DB3 "Found existing addon ${nsname_on_server}, basename=${basensname_on_server}"
+
+        # check if the addon is present in the directory and decide
+        # what to do with it
+        # this is not optimal because we're reading the files over and over
+        # again. But for small number of addons it doesn't matter so much.
+        found=0
+        for addon_path in ${addon_paths_in_files}; do
+            nsname_from_file=$(get-object-nsname-from-file ${addon_path})
+            if [[ "${nsname_from_file}" == "ERROR" ]]; then
+                log INFO "Cannot read object name from ${addon_path}. Ignoring"
+                continue
+            else
+                log DB2 "Found object name '${nsname_from_file}' in file ${addon_path}"
+            fi
+            suffix_from_file="$(get-suffix ${nsname_from_file} ${separator})"
+
+            log DB3 "matching: ${basensname_on_server}${suffix_from_file} == ${nsname_from_file}"
+            if [[ "${basensname_on_server}${suffix_from_file}" == "${nsname_from_file}" ]]; then
+                log DB3 "matched existing ${obj_type} ${nsname_on_server} to file ${addon_path}; suffix_on_server=${suffix_on_server}, suffix_from_file=${suffix_from_file}"
+                found=1
+                matched_files="${matched_files} ${addon_path}"
+                if [[ "${suffix_on_server}" == "${suffix_from_file}" ]]; then
+                    nsnames_for_ignore="${nsnames_for_ignore} ${nsname_from_file}"
+                else
+                    for_update="${for_update} ${nsname_on_server};${addon_path}"
+                fi
+                break
+            fi
+        done
+        if [[ ${found} -eq 0 ]]; then
+            log DB2 "No definition file found for replication controller ${nsname_on_server}. Scheduling for deletion"
+            nsnames_for_delete="${nsnames_for_delete} ${nsname_on_server}"
+        fi
+    done
+
+    log DB3 "matched_files=${matched_files}"
+
+
+    # note that if the addon file is invalid (or got removed after listing files
+    # but before we managed to match it) it will not be matched to any
+    # of the existing objects. So we will treat it as a new file
+    # and try to create its object.
+    for addon_path in ${addon_paths_in_files}; do
+        echo ${matched_files} | grep "${addon_path}" >/dev/null
+        if [[ $? -ne 0 ]]; then
+            new_files="${new_files} ${addon_path}"
+        fi
+    done
+}
+
+
+
+function reconcile-objects() {
+    local -r addon_path=$1
+    local -r obj_type=$2
+    local -r separator=$3    # name separator
+    match-objects ${addon_path} ${obj_type} ${separator}
+
+    log DBG "${obj_type}: nsnames_for_delete=${nsnames_for_delete}"
+    log DBG "${obj_type}: for_update=${for_update}"
+    log DBG "${obj_type}: nsnames_for_ignore=${nsnames_for_ignore}"
+    log DBG "${obj_type}: new_files=${new_files}"
+
+    delete-objects "${obj_type}" "${nsnames_for_delete}"
+    # wait for jobs below is a protection against changing the basename
+    # of a replication controllerm without changing the selector.
+    # If we don't wait, the new rc may be created before the old one is deleted
+    # In such case the old one will wait for all its pods to be gone, but the pods
+    # are created by the new replication controller.
+    # passing --cascade=false could solve the problem, but we want
+    # all orphan pods to be deleted.
+    wait-for-jobs
+    deleteResult=$?
+
+    create-objects "${obj_type}" "${new_files}"
+    update-objects "${obj_type}" "${for_update}"
+
+    local nsname
+    for nsname in ${nsnames_for_ignore}; do
+        log DB2 "The ${obj_type} ${nsname} is already up to date"
+    done
+
+    wait-for-jobs
+    createUpdateResult=$?
+
+    if [[ ${deleteResult} -eq 0 ]] && [[ ${createUpdateResult} -eq 0 ]]; then
+        return 0
+    else
+        return 1
+    fi
+}
+
+function update-addons() {
+    local -r addon_path=$1
+    # be careful, reconcile-objects uses global variables
+    reconcile-objects ${addon_path} ReplicationController "-" &
+    reconcile-objects ${addon_path} Deployment "-" &
+
+    # We don't expect names to be versioned for the following kinds, so
+    # we match the entire name, ignoring version suffix.
+    # That's why we pass an empty string as the version separator.
+    # If the description differs on disk, the object should be recreated.
+    # This is not implemented in this version.
+    reconcile-objects ${addon_path} Service "" &
+    reconcile-objects ${addon_path} PersistentVolume "" &
+    reconcile-objects ${addon_path} PersistentVolumeClaim "" &
+
+    wait-for-jobs
+    if [[ $? -eq 0 ]]; then
+        log INFO "== Kubernetes addon update completed successfully at $(date -Is) =="
+    else
+        log WRN "== Kubernetes addon update completed with errors at $(date -Is) =="
+    fi
+}
+
+# input parameters:
+# $1 input directory
+# $2 retry period in seconds - the script will retry api-server errors for approximately
+#     this amound of time (it is not very precise), at interval equal $DELAY_AFTER_ERROR_SEC.
+#
+
+if [[ $# -ne 2 ]]; then
+    echo "Illegal number of parameters. Usage $0 addon-dir [retry-period]" 1>&2
+    exit 1
+fi
+
+NUM_TRIES=$(($2 / ${DELAY_AFTER_ERROR_SEC}))
+if [[ ${NUM_TRIES} -le 0 ]]; then
+    NUM_TRIES=1
+fi
+
+addon_path=$1
+update-addons ${addon_path}
--- a/upup/models/nodeup/_kubernetes_master/kube-addons/files/etc/kubernetes/kube-addon-update.sh.meta
+++ b/upup/models/nodeup/_kubernetes_master/kube-addons/files/etc/kubernetes/kube-addon-update.sh.meta
@ -0,0 +1,3 @@
+{
+  "mode": "0755"
+}
--- a/upup/models/nodeup/_kubernetes_master/kube-addons/files/etc/kubernetes/kube-addons.sh
+++ b/upup/models/nodeup/_kubernetes_master/kube-addons/files/etc/kubernetes/kube-addons.sh
@ -0,0 +1,125 @@
+#!/bin/bash
+
+# Copyright 2014 The Kubernetes Authors All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# The business logic for whether a given object should be created
+# was already enforced by salt, and /etc/kubernetes/addons is the
+# managed result is of that. Start everything below that directory.
+KUBECTL=${KUBECTL_BIN:-/usr/local/bin/kubectl}
+
+ADDON_CHECK_INTERVAL_SEC=${TEST_ADDON_CHECK_INTERVAL_SEC:-600}
+
+SYSTEM_NAMESPACE=kube-system
+trusty_master=${TRUSTY_MASTER:-false}
+
+function ensure_python() {
+  if ! python --version > /dev/null 2>&1; then    
+    echo "No python on the machine, will use a python image"
+    local -r PYTHON_IMAGE=gcr.io/google_containers/python:v1
+    export PYTHON="docker run --interactive --rm --net=none ${PYTHON_IMAGE} python"
+  else
+    export PYTHON=python
+  fi
+}
+
+# $1 filename of addon to start.
+# $2 count of tries to start the addon.
+# $3 delay in seconds between two consecutive tries
+# $4 namespace
+function start_addon() {
+  local -r addon_filename=$1;
+  local -r tries=$2;
+  local -r delay=$3;
+  local -r namespace=$4
+
+  create-resource-from-string "$(cat ${addon_filename})" "${tries}" "${delay}" "${addon_filename}" "${namespace}"
+}
+
+# $1 string with json or yaml.
+# $2 count of tries to start the addon.
+# $3 delay in seconds between two consecutive tries
+# $4 name of this object to use when logging about it.
+# $5 namespace for this object
+function create-resource-from-string() {
+  local -r config_string=$1;
+  local tries=$2;
+  local -r delay=$3;
+  local -r config_name=$4;
+  local -r namespace=$5;
+  while [ ${tries} -gt 0 ]; do
+    echo "${config_string}" | ${KUBECTL} --namespace="${namespace}" apply -f - && \
+        echo "== Successfully started ${config_name} in namespace ${namespace} at $(date -Is)" && \
+        return 0;
+    let tries=tries-1;
+    echo "== Failed to start ${config_name} in namespace ${namespace} at $(date -Is). ${tries} tries remaining. =="
+    sleep ${delay};
+  done
+  return 1;
+}
+
+# The business logic for whether a given object should be created
+# was already enforced by salt, and /etc/kubernetes/addons is the
+# managed result is of that. Start everything below that directory.
+echo "== Kubernetes addon manager started at $(date -Is) with ADDON_CHECK_INTERVAL_SEC=${ADDON_CHECK_INTERVAL_SEC} =="
+
+ensure_python
+
+# Load the kube-env, which has all the environment variables we care
+# about, in a flat yaml format.
+kube_env_yaml="/var/cache/kubernetes-install/kube_env.yaml"
+if [ ! -e "${kubelet_kubeconfig_file}" ]; then
+  eval $(${PYTHON} -c '''
+import pipes,sys,yaml
+
+for k,v in yaml.load(sys.stdin).iteritems():
+  print("readonly {var}={value}".format(var = k, value = pipes.quote(str(v))))
+''' < "${kube_env_yaml}")
+fi
+
+
+# Create the namespace that will be used to host the cluster-level add-ons.
+start_addon /etc/kubernetes/addons/namespace.yaml 100 10 "" &
+
+# Wait for the default service account to be created in the kube-system namespace.
+token_found=""
+while [ -z "${token_found}" ]; do
+  sleep .5
+  token_found=$(${KUBECTL} get --namespace="${SYSTEM_NAMESPACE}" serviceaccount default -o go-template="{{with index .secrets 0}}{{.name}}{{end}}" || true)
+done
+
+echo "== default service account in the ${SYSTEM_NAMESPACE} namespace has token ${token_found} =="
+
+# Create admission_control objects if defined before any other addon services. If the limits
+# are defined in a namespace other than default, we should still create the limits for the
+# default namespace.
+for obj in $(find /etc/kubernetes/admission-controls \( -name \*.yaml -o -name \*.json \)); do
+  start_addon "${obj}" 100 10 default &
+  echo "++ obj ${obj} is created ++"
+done
+
+# Check if the configuration has changed recently - in case the user
+# created/updated/deleted the files on the master.
+while true; do
+  start_sec=$(date +"%s")
+  #kube-addon-update.sh must be deployed in the same directory as this file
+  `dirname $0`/kube-addon-update.sh /etc/kubernetes/addons ${ADDON_CHECK_INTERVAL_SEC}
+  end_sec=$(date +"%s")
+  len_sec=$((${end_sec}-${start_sec}))
+  # subtract the time passed from the sleep time
+  if [[ ${len_sec} -lt ${ADDON_CHECK_INTERVAL_SEC} ]]; then
+    sleep_time=$((${ADDON_CHECK_INTERVAL_SEC}-${len_sec}))
+    sleep ${sleep_time}
+  fi
+done
--- a/upup/models/nodeup/_kubernetes_master/kube-addons/files/etc/kubernetes/kube-addons.sh.meta
+++ b/upup/models/nodeup/_kubernetes_master/kube-addons/files/etc/kubernetes/kube-addons.sh.meta
@ -0,0 +1,3 @@
+{
+  "mode": "0755"
+}
--- a/upup/models/nodeup/_kubernetes_master/kube-addons/services/kube-addons.service
+++ b/upup/models/nodeup/_kubernetes_master/kube-addons/services/kube-addons.service
@ -0,0 +1,9 @@
+[Unit]
+Description=Kubernetes Addon Object Manager
+Documentation=https://github.com/kubernetes/kubernetes
+
+[Service]
+ExecStart=/etc/kubernetes/kube-addons.sh
+
+[Install]
+WantedBy=multi-user.target
--- a/upup/models/nodeup/_kubernetes_master/kube-apiserver/files/etc/kubernetes/manifests/kube-apiserver.manifest.template
+++ b/upup/models/nodeup/_kubernetes_master/kube-apiserver/files/etc/kubernetes/manifests/kube-apiserver.manifest.template
@ -0,0 +1,100 @@
+{
+"apiVersion": "v1",
+"kind": "Pod",
+"metadata": {
+  "name":"kube-apiserver",
+  "namespace": "kube-system"
+},
+"spec":{
+"hostNetwork": true,
+"containers":[
+    {
+    "name": "kube-apiserver",
+    "image": "{{ .APIServer.Image }}",
+    "resources": {
+      "requests": {
+        "cpu": "250m"
+      }
+    },
+    "command": [
+                 "/bin/sh",
+                 "-c",
+                 "/usr/local/bin/kube-apiserver {{ BuildFlags .APIServer }} 1>>/var/log/kube-apiserver.log 2>&1"
+               ],
+    "livenessProbe": {
+      "httpGet": {
+        "host": "127.0.0.1",
+        "port": 8080,
+        "path": "/healthz"
+      },
+      "initialDelaySeconds": 15,
+      "timeoutSeconds": 15
+    },
+    "ports":[
+      { "name": "https",
+        "containerPort": {{ .APIServer.SecurePort }},
+        "hostPort": {{ .APIServer.SecurePort }} },{
+       "name": "local",
+        "containerPort": 8080,
+        "hostPort": 8080}
+        ],
+    "volumeMounts": [
+        {"name": "usrsharessl","mountPath": "/usr/share/ssl", "readOnly": true}, {"name": "usrssl","mountPath": "/usr/ssl", "readOnly": true}, {"name": "usrlibssl","mountPath": "/usr/lib/ssl", "readOnly": true}, {"name": "usrlocalopenssl","mountPath": "/usr/local/openssl", "readOnly": true},
+
+        { "name": "srvkube",
+        "mountPath": "{{ .APIServer.PathSrvKubernetes }}",
+        "readOnly": true},
+        { "name": "logfile",
+        "mountPath": "/var/log/kube-apiserver.log",
+        "readOnly": false},
+        { "name": "etcssl",
+        "mountPath": "/etc/ssl",
+        "readOnly": true},
+        { "name": "varssl",
+        "mountPath": "/var/ssl",
+        "readOnly": true},
+        { "name": "etcopenssl",
+        "mountPath": "/etc/openssl",
+        "readOnly": true},
+        { "name": "etcpkitls",
+        "mountPath": "/etc/pki/tls",
+        "readOnly": true},
+        { "name": "srvsshproxy",
+        "mountPath": "{{ .APIServer.PathSrvSshproxy }}",
+        "readOnly": false}
+      ]
+    }
+],
+"volumes":[
+  {"name": "usrsharessl","hostPath": {"path": "/usr/share/ssl"}}, {"name": "usrssl","hostPath": {"path": "/usr/ssl"}}, {"name": "usrlibssl","hostPath": {"path": "/usr/lib/ssl"}}, {"name": "usrlocalopenssl","hostPath": {"path": "/usr/local/openssl"}},
+
+  { "name": "srvkube",
+    "hostPath": {
+        "path": "{{ .APIServer.PathSrvKubernetes }}"}
+  },
+  { "name": "logfile",
+    "hostPath": {
+        "path": "/var/log/kube-apiserver.log"}
+  },
+  { "name": "etcssl",
+    "hostPath": {
+        "path": "/etc/ssl"}
+  },
+  { "name": "varssl",
+    "hostPath": {
+        "path": "/var/ssl"}
+  },
+  { "name": "etcopenssl",
+    "hostPath": {
+        "path": "/etc/openssl"}
+  },
+  { "name": "etcpkitls",
+    "hostPath": {
+        "path": "/etc/pki/tls"}
+  },
+  { "name": "srvsshproxy",
+    "hostPath": {
+        "path": "{{ .APIServer.PathSrvSshproxy }}"}
+  }
+]
+}}
--- a/upup/models/nodeup/_kubernetes_master/kube-apiserver/files/srv/kubernetes/basic_auth.csv.template
+++ b/upup/models/nodeup/_kubernetes_master/kube-apiserver/files/srv/kubernetes/basic_auth.csv.template
@ -0,0 +1 @@
+{{ .KubePassword }},{{ .KubeUser }},admin
--- a/upup/models/nodeup/_kubernetes_master/kube-apiserver/files/srv/kubernetes/basic_auth.csv.template.meta
+++ b/upup/models/nodeup/_kubernetes_master/kube-apiserver/files/srv/kubernetes/basic_auth.csv.template.meta
@ -0,0 +1,3 @@
+{
+  "mode": "0600"
+}
--- a/upup/models/nodeup/_kubernetes_master/kube-apiserver/files/srv/kubernetes/known_tokens.csv.template
+++ b/upup/models/nodeup/_kubernetes_master/kube-apiserver/files/srv/kubernetes/known_tokens.csv.template
@ -0,0 +1,3 @@
+{{ range $id, $token := .Tokens }}
+{{ $token }},{{ $id }},{{ $id }}
+{{ end }}
--- a/upup/models/nodeup/_kubernetes_master/kube-apiserver/files/srv/kubernetes/known_tokens.csv.template.meta
+++ b/upup/models/nodeup/_kubernetes_master/kube-apiserver/files/srv/kubernetes/known_tokens.csv.template.meta
@ -0,0 +1,3 @@
+{
+  "mode": "0600"
+}
--- a/upup/models/nodeup/_kubernetes_master/kube-apiserver/files/var/log/kube-apiserver.log
+++ b/upup/models/nodeup/_kubernetes_master/kube-apiserver/files/var/log/kube-apiserver.log
--- a/upup/models/nodeup/_kubernetes_master/kube-apiserver/files/var/log/kube-apiserver.log.meta
+++ b/upup/models/nodeup/_kubernetes_master/kube-apiserver/files/var/log/kube-apiserver.log.meta
@ -0,0 +1,3 @@
+{
+  "ifNotExists": true
+}
--- a/upup/models/nodeup/_kubernetes_master/kube-apiserver/options/_aws/kube-apiserver.aws
+++ b/upup/models/nodeup/_kubernetes_master/kube-apiserver/options/_aws/kube-apiserver.aws
@ -0,0 +1,2 @@
+APIServer:
+  CloudProvider: aws
--- a/upup/models/nodeup/_kubernetes_master/kube-apiserver/options/_gce/kube-apiserver.gce
+++ b/upup/models/nodeup/_kubernetes_master/kube-apiserver/options/_gce/kube-apiserver.gce
@ -0,0 +1,2 @@
+APIServer:
+  CloudProvider: gce
--- a/upup/models/nodeup/_kubernetes_master/kube-apiserver/options/kube-apiserver
+++ b/upup/models/nodeup/_kubernetes_master/kube-apiserver/options/kube-apiserver
@ -0,0 +1,17 @@
+APIServer:
+  SecurePort: 443
+  PathSrvKubernetes: /srv/kubernetes
+  PathSrvSshproxy: /srv/sshproxy
+  Image: gcr.io/google_containers/kube-apiserver:v1.2.2
+  Address: 127.0.0.1
+  EtcdServers: http://127.0.0.1:4001
+  EtcdServersOverrides: /events#http://127.0.0.1:4002
+  AdmissionControl: NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,PersistentVolumeLabel
+  ServiceClusterIPRange: 10.0.0.0/16
+  ClientCAFile: /srv/kubernetes/ca.crt
+  BasicAuthFile: /srv/kubernetes/basic_auth.csv
+  TLSCertFile: /srv/kubernetes/server.cert
+  TLSPrivateKeyFile: /srv/kubernetes/server.key
+  TokenAuthFile: /srv/kubernetes/known_tokens.csv
+  LogLevel: 2
+  AllowPrivileged: true
--- a/upup/models/nodeup/_kubernetes_master/kube-controller-manager/files/etc/kubernetes/manifests/kube-controller-manager.template
+++ b/upup/models/nodeup/_kubernetes_master/kube-controller-manager/files/etc/kubernetes/manifests/kube-controller-manager.template
@ -0,0 +1,84 @@
+{
+"apiVersion": "v1",
+"kind": "Pod",
+"metadata": {
+  "name":"kube-controller-manager",
+  "namespace": "kube-system"
+},
+"spec":{
+"hostNetwork": true,
+"containers":[
+    {
+    "name": "kube-controller-manager",
+    "image": "{{ .KubeControllerManager.Image }}",
+    "resources": {
+      "requests": {
+        "cpu": "200m"
+      }
+    },
+    "command": [
+                 "/bin/sh",
+                 "-c",
+                 "/usr/local/bin/kube-controller-manager {{ BuildFlags .KubeControllerManager }} 1>>/var/log/kube-controller-manager.log 2>&1"
+               ],
+    "livenessProbe": {
+      "httpGet": {
+        "host": "127.0.0.1",
+        "port": 10252,
+        "path": "/healthz"
+      },
+      "initialDelaySeconds": 15,
+      "timeoutSeconds": 15
+    },
+    "volumeMounts": [
+        {"name": "usrsharessl","mountPath": "/usr/share/ssl", "readOnly": true}, {"name": "usrssl","mountPath": "/usr/ssl", "readOnly": true}, {"name": "usrlibssl","mountPath": "/usr/lib/ssl", "readOnly": true}, {"name": "usrlocalopenssl","mountPath": "/usr/local/openssl", "readOnly": true},
+        { "name": "srvkube",
+        "mountPath": "{{ .KubeControllerManager.PathSrvKubernetes }}",
+        "readOnly": true},
+        { "name": "logfile",
+        "mountPath": "/var/log/kube-controller-manager.log",
+        "readOnly": false},
+        { "name": "etcssl",
+        "mountPath": "/etc/ssl",
+        "readOnly": true},
+        { "name": "varssl",
+        "mountPath": "/var/ssl",
+        "readOnly": true},
+        { "name": "etcopenssl",
+        "mountPath": "/etc/openssl",
+        "readOnly": true},
+        { "name": "etcpkitls",
+        "mountPath": "/etc/pki/tls",
+        "readOnly": true}
+      ]
+    }
+],
+"volumes":[
+  {"name": "usrsharessl","hostPath": {"path": "/usr/share/ssl"}}, {"name": "usrssl","hostPath": {"path": "/usr/ssl"}}, {"name": "usrlibssl","hostPath": {"path": "/usr/lib/ssl"}}, {"name": "usrlocalopenssl","hostPath": {"path": "/usr/local/openssl"}},
+
+  { "name": "srvkube",
+    "hostPath": {
+        "path": "{{ .KubeControllerManager.PathSrvKubernetes }}"}
+  },
+  { "name": "logfile",
+    "hostPath": {
+        "path": "/var/log/kube-controller-manager.log"}
+  },
+  { "name": "etcssl",
+    "hostPath": {
+        "path": "/etc/ssl"}
+  },
+  { "name": "varssl",
+    "hostPath": {
+        "path": "/var/ssl"}
+  },
+  { "name": "etcopenssl",
+    "hostPath": {
+        "path": "/etc/openssl"}
+  },
+  { "name": "etcpkitls",
+    "hostPath": {
+        "path": "/etc/pki/tls"}
+  }
+]
+}}
--- a/upup/models/nodeup/_kubernetes_master/kube-controller-manager/files/var/log/kube-controller-manager.log
+++ b/upup/models/nodeup/_kubernetes_master/kube-controller-manager/files/var/log/kube-controller-manager.log
--- a/upup/models/nodeup/_kubernetes_master/kube-controller-manager/files/var/log/kube-controller-manager.log.meta
+++ b/upup/models/nodeup/_kubernetes_master/kube-controller-manager/files/var/log/kube-controller-manager.log.meta
@ -0,0 +1,3 @@
+{
+  "ifNotExists": true
+}
--- a/upup/models/nodeup/_kubernetes_master/kube-controller-manager/options/_aws/kube-controller-manager.aws
+++ b/upup/models/nodeup/_kubernetes_master/kube-controller-manager/options/_aws/kube-controller-manager.aws
@ -0,0 +1,2 @@
+KubeControllerManager:
+  CloudProvider: aws
--- a/upup/models/nodeup/_kubernetes_master/kube-controller-manager/options/_gce/kube-controller-manager.gce
+++ b/upup/models/nodeup/_kubernetes_master/kube-controller-manager/options/_gce/kube-controller-manager.gce
@ -0,0 +1,2 @@
+KubeControllerManager:
+  CloudProvider: gce
--- a/upup/models/nodeup/_kubernetes_master/kube-controller-manager/options/kube-controller-manager
+++ b/upup/models/nodeup/_kubernetes_master/kube-controller-manager/options/kube-controller-manager
@ -0,0 +1,10 @@
+KubeControllerManager:
+  PathSrvKubernetes: /srv/kubernetes
+  Image: gcr.io/google_containers/kube-controller-manager:v1.2.2
+  Master: 127.0.0.1:8080
+  ClusterName: kubernetes
+  ClusterCIDR: 10.244.0.0/16
+  AllocateNodeCIDRs: true
+  ServiceAccountPrivateKeyFile: /srv/kubernetes/server.key
+  LogLevel: 2
+  RootCAFile: /srv/kubernetes/ca.crt
--- a/upup/models/nodeup/_kubernetes_master/kube-dns/files/etc/kubernetes/addons/dns/skydns-rc.yaml.template
+++ b/upup/models/nodeup/_kubernetes_master/kube-dns/files/etc/kubernetes/addons/dns/skydns-rc.yaml.template
@ -0,0 +1,119 @@
+apiVersion: v1
+kind: ReplicationController
+metadata:
+  name: kube-dns-v10
+  namespace: kube-system
+  labels:
+    k8s-app: kube-dns
+    version: v10
+    kubernetes.io/cluster-service: "true"
+spec:
+  replicas: {{ .DNS.Replicas }}
+  selector:
+    k8s-app: kube-dns
+    version: v10
+  template:
+    metadata:
+      labels:
+        k8s-app: kube-dns
+        version: v10
+        kubernetes.io/cluster-service: "true"
+    spec:
+      containers:
+      - name: etcd
+        image: gcr.io/google_containers/etcd:2.0.9
+        resources:
+          # keep request = limit to keep this container in guaranteed class
+          limits:
+            cpu: 100m
+            memory: 50Mi
+          requests:
+            cpu: 100m
+            memory: 50Mi
+        command:
+        - /usr/local/bin/etcd
+        - -data-dir
+        - /var/etcd/data
+        - -listen-client-urls
+        - http://127.0.0.1:2379,http://127.0.0.1:4001
+        - -advertise-client-urls
+        - http://127.0.0.1:2379,http://127.0.0.1:4001
+        - -initial-cluster-token
+        - skydns-etcd
+        volumeMounts:
+        - name: etcd-storage
+          mountPath: /var/etcd/data
+      - name: kube2sky
+        image: gcr.io/google_containers/kube2sky:1.12
+        resources:
+          # keep request = limit to keep this container in guaranteed class
+          limits:
+            cpu: 100m
+            memory: 50Mi
+          requests:
+            cpu: 100m
+            memory: 50Mi
+        command:
+        - /kube2sky
+        args:
+        - -domain={{ .DNS.Domain }}
+      - name: skydns
+        image: gcr.io/google_containers/skydns:2015-10-13-8c72f8c
+        resources:
+          # keep request = limit to keep this container in guaranteed class
+          limits:
+            cpu: 100m
+            memory: 50Mi
+          requests:
+            cpu: 100m
+            memory: 50Mi
+        command:
+        - /skydns
+        args:
+        - -machines=http://127.0.0.1:4001
+        - -addr=0.0.0.0:53
+        - -ns-rotate=false
+        - -domain={{ .DNS.Domain }}.
+        ports:
+        - containerPort: 53
+          name: dns
+          protocol: UDP
+        - containerPort: 53
+          name: dns-tcp
+          protocol: TCP
+        livenessProbe:
+          httpGet:
+            path: /healthz
+            port: 8080
+            scheme: HTTP
+          initialDelaySeconds: 30
+          timeoutSeconds: 5
+        readinessProbe:
+          httpGet:
+            path: /healthz
+            port: 8080
+            scheme: HTTP
+          initialDelaySeconds: 1
+          timeoutSeconds: 5
+      - name: healthz
+        image: gcr.io/google_containers/exechealthz:1.0
+        resources:
+          # keep request = limit to keep this container in guaranteed class
+          limits:
+            cpu: 10m
+            memory: 20Mi
+          requests:
+            cpu: 10m
+            memory: 20Mi
+        command:
+        - /exechealthz
+        args:
+        - -cmd=nslookup kubernetes.default.svc.{{ .DNS.Domain }} 127.0.0.1 >/dev/null
+        - -port=8080
+        ports:
+        - containerPort: 8080
+          protocol: TCP
+      volumes:
+      - name: etcd-storage
+        emptyDir: {}
+      dnsPolicy: Default  # Don't use cluster DNS.
--- a/upup/models/nodeup/_kubernetes_master/kube-dns/files/etc/kubernetes/addons/dns/skydns-svc.yaml.template
+++ b/upup/models/nodeup/_kubernetes_master/kube-dns/files/etc/kubernetes/addons/dns/skydns-svc.yaml.template
@ -0,0 +1,20 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: kube-dns
+  namespace: kube-system
+  labels:
+    k8s-app: kube-dns
+    kubernetes.io/cluster-service: "true"
+    kubernetes.io/name: "KubeDNS"
+spec:
+  selector:
+    k8s-app: kube-dns
+  clusterIP:  {{ .DNS.ServerIP }}
+  ports:
+  - name: dns
+    port: 53
+    protocol: UDP
+  - name: dns-tcp
+    port: 53
+    protocol: TCP
--- a/upup/models/nodeup/_kubernetes_master/kube-dns/options/kube-dns
+++ b/upup/models/nodeup/_kubernetes_master/kube-dns/options/kube-dns
@ -0,0 +1,4 @@
+DNS:
+  Replicas: 1
+  ServerIP: 10.0.0.10
+  Domain: cluster.local
--- a/upup/models/nodeup/_kubernetes_master/kube-scheduler/files/etc/kubernetes/manifests/kube-scheduler.template
+++ b/upup/models/nodeup/_kubernetes_master/kube-scheduler/files/etc/kubernetes/manifests/kube-scheduler.template
@ -0,0 +1,48 @@
+{
+"apiVersion": "v1",
+"kind": "Pod",
+"metadata": {
+  "name":"kube-scheduler",
+  "namespace": "kube-system"
+},
+"spec":{
+"hostNetwork": true,
+"containers":[
+    {
+    "name": "kube-scheduler",
+    "image": "{{ .KubeScheduler.Image }}",
+    "resources": {
+      "requests": {
+        "cpu": "100m"
+      }
+    },
+    "command": [
+                 "/bin/sh",
+                 "-c",
+                 "/usr/local/bin/kube-scheduler {{ BuildFlags .KubeScheduler }} 1>>/var/log/kube-scheduler.log 2>&1"
+               ],
+    "livenessProbe": {
+      "httpGet": {
+        "host": "127.0.0.1",
+        "port": 10251,
+        "path": "/healthz"
+      },
+      "initialDelaySeconds": 15,
+      "timeoutSeconds": 15
+    },
+    "volumeMounts": [
+        {
+          "name": "logfile",
+          "mountPath": "/var/log/kube-scheduler.log",
+          "readOnly": false
+        }
+      ]
+    }
+],
+"volumes":[
+  { "name": "logfile",
+    "hostPath": {
+        "path": "/var/log/kube-scheduler.log"}
+  }
+]
+}}
--- a/upup/models/nodeup/_kubernetes_master/kube-scheduler/files/var/log/kube-scheduler.log
+++ b/upup/models/nodeup/_kubernetes_master/kube-scheduler/files/var/log/kube-scheduler.log
--- a/upup/models/nodeup/_kubernetes_master/kube-scheduler/files/var/log/kube-scheduler.log.meta
+++ b/upup/models/nodeup/_kubernetes_master/kube-scheduler/files/var/log/kube-scheduler.log.meta
@ -0,0 +1,3 @@
+{
+  "ifNotExists": true
+}
--- a/upup/models/nodeup/_kubernetes_master/kube-scheduler/options/kube-scheduler
+++ b/upup/models/nodeup/_kubernetes_master/kube-scheduler/options/kube-scheduler
@ -0,0 +1,4 @@
+KubeScheduler:
+  Image: gcr.io/google_containers/kube-scheduler:v1.2.2
+  Master: 127.0.0.1:8080
+  LogLevel: 2
--- a/upup/models/nodeup/_kubernetes_pool/_kube-node-unpacker/files/etc/kubernetes/kube-node-unpacker.sh
+++ b/upup/models/nodeup/_kubernetes_pool/_kube-node-unpacker/files/etc/kubernetes/kube-node-unpacker.sh
@ -0,0 +1,46 @@
+#!/bin/bash
+
+# Copyright 2015 The Kubernetes Authors All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# loadedImageFlags is a bit-flag to track which docker images loaded successfully.
+let loadedImageFlags=0
+
+while true; do
+  restart_docker=false
+
+  if which docker 1>/dev/null 2>&1; then
+
+    timeout 30 docker load -i /srv/salt/kube-bins/kube-proxy.tar 1>/dev/null 2>&1
+    rc=$?
+    if [[ "${rc}" == 0 ]]; then
+      let loadedImageFlags="${loadedImageFlags}|1"
+    elif [[ "${rc}" == 124 ]]; then
+      restart_docker=true
+    fi
+  fi
+
+  # required docker images got installed. exit while loop.
+  if [[ "${loadedImageFlags}" == 1 ]]; then break; fi
+
+  # Sometimes docker load hang, restart docker daemon resolve the issue
+  if [[ "${restart_docker}" ]]; then service docker restart; fi
+
+  # sleep for 15 seconds before attempting to load docker images again
+  sleep 15
+
+done
+
+# Now exit. After kube-push, salt will notice that the service is down and it
+# will start it and new docker images will be loaded.
--- a/upup/models/nodeup/_kubernetes_pool/_kube-node-unpacker/files/etc/kubernetes/kube-node-unpacker.sh.meta
+++ b/upup/models/nodeup/_kubernetes_pool/_kube-node-unpacker/files/etc/kubernetes/kube-node-unpacker.sh.meta
@ -0,0 +1,3 @@
+{
+  "mode": "0755"
+}
--- a/upup/models/nodeup/_kubernetes_pool/_kube-node-unpacker/files/srv/salt/kube-bins/kube-proxy.tar.asset
+++ b/upup/models/nodeup/_kubernetes_pool/_kube-node-unpacker/files/srv/salt/kube-bins/kube-proxy.tar.asset
@ -0,0 +1,2 @@
+{
+}
--- a/upup/models/nodeup/_kubernetes_pool/_kube-node-unpacker/services/kube-node-unpacker.service
+++ b/upup/models/nodeup/_kubernetes_pool/_kube-node-unpacker/services/kube-node-unpacker.service
@ -0,0 +1,9 @@
+[Unit]
+Description=Kubernetes Node Unpacker
+Documentation=https://github.com/kubernetes/kubernetes
+
+[Service]
+ExecStart=/etc/kubernetes/kube-node-unpacker.sh
+
+[Install]
+WantedBy=multi-user.target
--- a/upup/models/nodeup/_kubernetes_pool/helpers/_aws/files/usr/share/google/safe_format_and_mount
+++ b/upup/models/nodeup/_kubernetes_pool/helpers/_aws/files/usr/share/google/safe_format_and_mount
@ -0,0 +1,147 @@
+#! /bin/bash
+# Copyright 2013 Google Inc. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Mount a disk, formatting it if necessary.  If the disk looks like it may
+# have been formatted before, we will not format it.
+#
+# This script uses blkid and file to search for magic "formatted" bytes
+# at the beginning of the disk.  Furthermore, it attempts to use fsck to
+# repair the filesystem before formatting it.
+
+FSCK=fsck.ext4
+MOUNT_OPTIONS="discard,defaults"
+MKFS="mkfs.ext4 -E lazy_itable_init=0,lazy_journal_init=0 -F"
+if [ -e /etc/redhat-release ]; then
+    if grep -q '6\..' /etc/redhat-release; then
+      # lazy_journal_init is not recognized in redhat 6
+      MKFS="mkfs.ext4 -E lazy_itable_init=0 -F"
+    elif grep -q '7\..' /etc/redhat-release; then
+      FSCK=fsck.xfs
+      MKFS=mkfs.xfs
+    fi
+fi
+
+LOGTAG=safe_format_and_mount
+LOGFACILITY=user
+
+function log() {
+  local readonly severity=$1; shift;
+  logger -t ${LOGTAG} -p ${LOGFACILITY}.${severity} -s "$@"
+}
+
+function log_command() {
+  local readonly log_file=$(mktemp)
+  local readonly retcode
+  log info "Running: $*"
+  $* > ${log_file} 2>&1
+  retcode=$?
+  # only return the last 1000 lines of the logfile, just in case it's HUGE.
+  tail -1000 ${log_file} | logger -t ${LOGTAG} -p ${LOGFACILITY}.info -s
+  rm -f ${log_file}
+  return ${retcode}
+}
+
+function help() {
+  cat >&2 <<EOF
+$0 [-f fsck_cmd] [-m mkfs_cmd] [-o mount_opts] <device> <mountpoint>
+EOF
+  exit 0
+}
+
+while getopts ":hf:o:m:" opt; do
+  case $opt in
+    h) help;;
+    f) FSCK=$OPTARG;;
+    o) MOUNT_OPTIONS=$OPTARG;;
+    m) MKFS=$OPTARG;;
+    -) break;;
+    \?) log error "Invalid option: -${OPTARG}"; exit 1;;
+    :) log "Option -${OPTARG} requires an argument."; exit 1;;
+  esac
+done
+
+shift $(($OPTIND - 1))
+readonly DISK=$1
+readonly MOUNTPOINT=$2
+
+[[ -z ${DISK} ]] && help
+[[ -z ${MOUNTPOINT} ]] && help
+
+function disk_looks_unformatted() {
+  blkid ${DISK}
+  if [[ $? == 0 ]]; then
+    return 0
+  fi
+
+  local readonly file_type=$(file --special-files ${DISK})
+  case ${file_type} in
+    *filesystem*)
+      return 0;;
+  esac
+
+  return 1
+}
+
+function format_disk() {
+  log_command ${MKFS} ${DISK}
+}
+
+function try_repair_disk() {
+  log_command ${FSCK} -a ${DISK}
+  local readonly fsck_return=$?
+  if [[ ${fsck_return} -ge 8 ]]; then
+    log error "Fsck could not correct errors on ${DISK}"
+    return 1
+  fi
+  if [[ ${fsck_return} -gt 0 ]]; then
+    log warning "Fsck corrected errors on ${DISK}"
+  fi
+  return 0
+}
+
+function try_mount() {
+  local mount_retcode
+  try_repair_disk
+
+  log_command mount -o ${MOUNT_OPTIONS} ${DISK} ${MOUNTPOINT}
+  mount_retcode=$?
+  if [[ ${mount_retcode} == 0 ]]; then
+    return 0
+  fi
+
+  # Check to see if it looks like a filesystem before formatting it.
+  disk_looks_unformatted ${DISK}
+  if [[ $? == 0 ]]; then
+    log error "Disk ${DISK} looks formatted but won't mount.  Giving up."
+    return ${mount_retcode}
+  fi
+
+  # The disk looks like it's not been formatted before.
+  format_disk
+  if [[ $? != 0 ]]; then
+    log error "Format of ${DISK} failed."
+  fi
+
+  log_command mount -o ${MOUNT_OPTIONS} ${DISK} ${MOUNTPOINT}
+  mount_retcode=$?
+  if [[ ${mount_retcode} == 0 ]]; then
+    return 0
+  fi
+  log error "Tried everything we could, but could not mount ${DISK}."
+  return ${mount_retcode}
+}
+
+try_mount
+exit $?
--- a/upup/models/nodeup/_kubernetes_pool/helpers/_aws/files/usr/share/google/safe_format_and_mount.meta
+++ b/upup/models/nodeup/_kubernetes_pool/helpers/_aws/files/usr/share/google/safe_format_and_mount.meta
@ -0,0 +1,3 @@
+{
+  "mode": "0755"
+}
--- a/upup/models/nodeup/_kubernetes_pool/kube-proxy/files/etc/kubernetes/manifests/kube-proxy.manifest.template
+++ b/upup/models/nodeup/_kubernetes_pool/kube-proxy/files/etc/kubernetes/manifests/kube-proxy.manifest.template
@ -0,0 +1,40 @@
+# kube-proxy podspec
+apiVersion: v1
+kind: Pod
+metadata:
+  name: kube-proxy
+  namespace: kube-system
+spec:
+  hostNetwork: true
+  containers:
+  - name: kube-proxy
+    image: {{ .KubeProxy.Image }}
+    resources:
+      requests:
+        cpu: {{ .KubeProxy.CPURequest }}
+    command:
+    - /bin/sh
+    - -c
+    - kube-proxy --kubeconfig=/var/lib/kube-proxy/kubeconfig --resource-container="" {{ BuildFlags .KubeProxy }} 1>>/var/log/kube-proxy.log 2>&1
+    securityContext:
+      privileged: true
+    volumeMounts:
+    - mountPath: /etc/ssl/certs
+      name: ssl-certs-host
+      readOnly: true
+    - mountPath: /var/log
+      name: varlog
+      readOnly: false
+    - mountPath: /var/lib/kube-proxy/kubeconfig
+      name: kubeconfig
+      readOnly: false
+  volumes:
+  - hostPath:
+      path: /usr/share/ca-certificates
+    name: ssl-certs-host
+  - hostPath:
+      path: /var/lib/kube-proxy/kubeconfig
+    name: kubeconfig
+  - hostPath:
+      path: /var/log
+    name: varlog
--- a/upup/models/nodeup/_kubernetes_pool/kube-proxy/files/var/lib/kube-proxy/kubeconfig.template
+++ b/upup/models/nodeup/_kubernetes_pool/kube-proxy/files/var/lib/kube-proxy/kubeconfig.template
@ -0,0 +1,16 @@
+apiVersion: v1
+kind: Config
+users:
+- name: kube-proxy
+  user:
+    token: {{ .GetToken "kube-proxy" }}
+clusters:
+- name: local
+  cluster:
+    certificate-authority-data: {{ Base64Encode .CACertificate.AsString }}
+contexts:
+- context:
+    cluster: local
+    user: kube-proxy
+  name: service-account-context
+current-context: service-account-context
--- a/upup/models/nodeup/_kubernetes_pool/kube-proxy/files/var/lib/kube-proxy/kubeconfig.template.meta
+++ b/upup/models/nodeup/_kubernetes_pool/kube-proxy/files/var/lib/kube-proxy/kubeconfig.template.meta
@ -0,0 +1,3 @@
+{
+  "mode": "0400"
+}
--- a/upup/models/nodeup/_kubernetes_pool/kube-proxy/files/var/log/kube-proxy.log
+++ b/upup/models/nodeup/_kubernetes_pool/kube-proxy/files/var/log/kube-proxy.log
--- a/upup/models/nodeup/_kubernetes_pool/kube-proxy/files/var/log/kube-proxy.log.meta
+++ b/upup/models/nodeup/_kubernetes_pool/kube-proxy/files/var/log/kube-proxy.log.meta
@ -0,0 +1,3 @@
+{
+  "ifNotExists": true
+}
--- a/upup/models/nodeup/_kubernetes_pool/kube-proxy/options/kube-proxy
+++ b/upup/models/nodeup/_kubernetes_pool/kube-proxy/options/kube-proxy
@ -0,0 +1,10 @@
+KubeProxy:
+  Image: gcr.io/google_containers/kube-proxy:v1.2.2
+  Master: https://kubernetes-master
+  LogLevel: 2
+  # 20m might cause kube-proxy CPU starvation on full nodes, resulting in
+  # delayed service updates. But, giving it more would be a breaking change
+  # to the overhead requirements for existing clusters.
+  # Any change here should be accompanied by a proportional change in CPU
+  # requests of other per-node add-ons (e.g. fluentd).
+  CPURequest: 20m
--- a/upup/models/nodeup/auto-upgrades/_debian_family/files/etc/apt/apt.conf.d/20auto-upgrades
+++ b/upup/models/nodeup/auto-upgrades/_debian_family/files/etc/apt/apt.conf.d/20auto-upgrades
@ -0,0 +1,4 @@
+APT::Periodic::Update-Package-Lists "1";
+APT::Periodic::Unattended-Upgrade "1";
+
+APT::Periodic::AutocleanInterval "7";
--- a/upup/models/nodeup/auto-upgrades/_debian_family/packages/unattended-upgrades
+++ b/upup/models/nodeup/auto-upgrades/_debian_family/packages/unattended-upgrades
--- a/upup/models/nodeup/docker/_gce/files/etc/sysctl.d/99-ip_forward.conf
+++ b/upup/models/nodeup/docker/_gce/files/etc/sysctl.d/99-ip_forward.conf
@ -0,0 +1,2 @@
+# Kubernetes
+net.ipv4.ip_forward=1
--- a/upup/models/nodeup/docker/_gce/files/etc/sysctl.d/99-ip_forward.conf.meta
+++ b/upup/models/nodeup/docker/_gce/files/etc/sysctl.d/99-ip_forward.conf.meta
@ -0,0 +1,3 @@
+{
+  "onChangeExecute": [ "sysctl", "--system" ]
+}
--- a/upup/models/nodeup/docker/_systemd/files/etc/sysconfig/docker.template
+++ b/upup/models/nodeup/docker/_systemd/files/etc/sysconfig/docker.template
@ -0,0 +1,2 @@
+DOCKER_OPTS="{{ BuildFlags .Docker }}"
+DOCKER_NOFILE=1000000
--- a/upup/models/nodeup/docker/_systemd/files/opt/kubernetes/helpers/docker-healthcheck
+++ b/upup/models/nodeup/docker/_systemd/files/opt/kubernetes/helpers/docker-healthcheck
@ -0,0 +1,45 @@
+#!/bin/bash
+
+# Copyright 2015 The Kubernetes Authors All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This script is intended to be run periodically, to check the health
+# of docker.  If it detects a failure, it will restart docker using systemctl.
+
+if timeout 10 docker version > /dev/null; then
+  echo "docker healthy"
+  exit 0
+fi
+
+echo "docker failed"
+echo "Giving docker 30 seconds grace before restarting"
+sleep 30
+
+if timeout 10 docker version > /dev/null; then
+  echo "docker recovered"
+  exit 0
+fi
+
+echo "docker still down; triggering docker restart"
+systemctl restart docker
+
+echo "Waiting 60 seconds to give docker time to start"
+sleep 60
+
+if timeout 10 docker version > /dev/null; then
+  echo "docker recovered"
+  exit 0
+fi
+
+echo "docker still failing"
--- a/upup/models/nodeup/docker/_systemd/files/opt/kubernetes/helpers/docker-healthcheck.meta
+++ b/upup/models/nodeup/docker/_systemd/files/opt/kubernetes/helpers/docker-healthcheck.meta
@ -0,0 +1,3 @@
+{
+  "mode": "0755"
+}
--- a/upup/models/nodeup/docker/_systemd/files/opt/kubernetes/helpers/docker-prestart
+++ b/upup/models/nodeup/docker/_systemd/files/opt/kubernetes/helpers/docker-prestart
@ -0,0 +1,21 @@
+#!/bin/bash
+
+# Copyright 2015 The Kubernetes Authors All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This script is intended to be run before we start Docker.
+
+# cleanup docker network checkpoint to avoid running into known issue
+# of docker (https://github.com/docker/docker/issues/18283)
+rm -rf /var/lib/docker/network
--- a/upup/models/nodeup/docker/_systemd/files/opt/kubernetes/helpers/docker-prestart.meta
+++ b/upup/models/nodeup/docker/_systemd/files/opt/kubernetes/helpers/docker-prestart.meta
@ -0,0 +1,3 @@
+{
+  "mode": "0755"
+}
--- a/upup/models/nodeup/docker/_systemd/services/docker-healthcheck.service
+++ b/upup/models/nodeup/docker/_systemd/services/docker-healthcheck.service
@ -0,0 +1,9 @@
+[Unit]
+Description=Run docker-healthcheck once
+
+[Service]
+Type=oneshot
+ExecStart=/opt/kubernetes/helpers/docker-healthcheck
+
+[Install]
+WantedBy=multi-user.target
--- a/upup/models/nodeup/docker/_systemd/services/docker-healthcheck.service.meta
+++ b/upup/models/nodeup/docker/_systemd/services/docker-healthcheck.service.meta
@ -0,0 +1,3 @@
+{
+  "manageState": false
+}
--- a/upup/models/nodeup/docker/_systemd/services/docker-healthcheck.timer
+++ b/upup/models/nodeup/docker/_systemd/services/docker-healthcheck.timer
@ -0,0 +1,9 @@
+[Unit]
+Description=Trigger docker-healthcheck periodically
+
+[Timer]
+OnUnitInactiveSec=10s
+Unit=docker-healthcheck.service
+
+[Install]
+WantedBy=multi-user.target
--- a/upup/models/nodeup/docker/_systemd/services/docker.service
+++ b/upup/models/nodeup/docker/_systemd/services/docker.service
@ -0,0 +1,21 @@
+[Unit]
+Description=Docker Application Container Engine
+Documentation=https://docs.docker.com
+After=network.target docker.socket
+Requires=docker.socket
+
+[Service]
+Type=notify
+EnvironmentFile=/etc/sysconfig/docker
+ExecStart=/usr/bin/docker daemon -H fd:// "$DOCKER_OPTS"
+MountFlags=slave
+LimitNOFILE=1048576
+LimitNPROC=1048576
+LimitCORE=infinity
+Restart=always
+RestartSec=2s
+StartLimitInterval=0
+ExecStartPre=/opt/kubernetes/helpers/docker-prestart
+
+[Install]
+WantedBy=multi-user.target
--- a/upup/models/nodeup/docker/_systemd/services/docker.service.meta
+++ b/upup/models/nodeup/docker/_systemd/services/docker.service.meta
@ -0,0 +1,3 @@
+{
+  "manageState": false
+}
--- a/upup/models/nodeup/docker/files/usr/share/doc/docker/apache.txt
+++ b/upup/models/nodeup/docker/files/usr/share/doc/docker/apache.txt
@ -0,0 +1,4 @@
+{
+  "source": "https://storage.googleapis.com/kubernetes-release/docker/apache2.txt",
+  "hash": "2b8b815229aa8a61e483fb4ba0588b8b6c491890"
+}
--- a/upup/models/nodeup/docker/options/_e2e_storage_test_environment/e2e.options
+++ b/upup/models/nodeup/docker/options/_e2e_storage_test_environment/e2e.options
@ -0,0 +1,2 @@
+Docker:
+  Storage: devicemapper
--- a/upup/models/nodeup/docker/options/_kubenet/kubenet.options
+++ b/upup/models/nodeup/docker/options/_kubenet/kubenet.options
@ -0,0 +1,5 @@
+{% set log_level = "--log-level=warn" -%}
+{% if pillar['docker_test_log_level'] is defined -%}
+  {% set log_level = pillar['docker_test_log_level'] -%}
+{% endif -%}
+docker.bridge=
--- a/upup/models/nodeup/docker/options/default
+++ b/upup/models/nodeup/docker/options/default
@ -0,0 +1,5 @@
+Docker:
+  Bridge: cbr0
+  LogLevel: warn
+  IPTables: false
+  IPMasq: false
--- a/upup/models/nodeup/docker/packages/_jessie/docker-engine
+++ b/upup/models/nodeup/docker/packages/_jessie/docker-engine
@ -0,0 +1,7 @@
+{
+  "version": "1.9.1-0~jessie",
+  "source": "http://apt.dockerproject.org/repo/pool/main/d/docker-engine/docker-engine_1.9.1-0~jessie_amd64.deb",
+  "hash": "c58c39008fd6399177f6b2491222e4438f518d78",
+
+  "preventStart": true
+}
--- a/upup/models/nodeup/docker/packages/bridge-utils
+++ b/upup/models/nodeup/docker/packages/bridge-utils
--- a/upup/models/nodeup/docker/packages/libapparmor1
+++ b/upup/models/nodeup/docker/packages/libapparmor1
--- a/upup/models/nodeup/docker/packages/perl
+++ b/upup/models/nodeup/docker/packages/perl
--- a/upup/models/nodeup/kube-client-tools/files/usr/local/bin/kubectl.asset
+++ b/upup/models/nodeup/kube-client-tools/files/usr/local/bin/kubectl.asset
@ -0,0 +1,2 @@
+{
+}
--- a/upup/models/nodeup/kube-client-tools/files/usr/local/bin/kubectl.asset.meta
+++ b/upup/models/nodeup/kube-client-tools/files/usr/local/bin/kubectl.asset.meta
@ -0,0 +1,3 @@
+{
+  "mode": "0755"
+}
--- a/upup/models/nodeup/kubelet/files/etc/sysconfig/kubelet.template
+++ b/upup/models/nodeup/kubelet/files/etc/sysconfig/kubelet.template
@ -0,0 +1 @@
+DAEMON_ARGS="{{ BuildFlags .Kubelet }}"
--- a/upup/models/nodeup/kubelet/files/usr/local/bin/kubelet.asset
+++ b/upup/models/nodeup/kubelet/files/usr/local/bin/kubelet.asset
@ -0,0 +1,2 @@
+{
+}
--- a/upup/models/nodeup/kubelet/files/usr/local/bin/kubelet.asset.meta
+++ b/upup/models/nodeup/kubelet/files/usr/local/bin/kubelet.asset.meta
@ -0,0 +1,3 @@
+{
+  "mode": "0755"
+}
--- a/Show More
+++ b/Show More
				`@ -0,0 +1 @@`
				`{{ BuildNodeConfig "cloudinit" "resources/config.yaml.template" Args }}`