kubernetes
/
community
mirror of https://github.com/kubernetes/community.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #254 from jessfraz/sip 

[Proposal] Pod Injection Policy
This commit is contained in:
Paul Morie 2017-02-23 01:34:37 -05:00 committed by GitHub
parent 36941645a3 a612a220cb
commit 2b56d8bcde
1 changed files with 718 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

718
contributors/design-proposals/pod-injection-policy.md Normal file
View File

@ -0,0 +1,718 @@
# Pod Injection Policy
* [Abstract](#abstract)
* [Motivation](#motivation)
* [Constraints and Assumptions](#constraints-and-assumptions)
* [Use Cases](#use-cases)
* [Summary](#summary)
* [Prior Art](#prior-art)
* [Objectives](#objectives)
* [Proposed Changes](#proposed-changes)
* [PodInjectionPolicy API object](#podinjectionpolicy-api-object)
* [Validations](#validations)
* [AdmissionControl Plug-in: PodInjectionPolicy](#admissioncontrol-plug-in-podinjectionpolicy)
* [Behavior](#behavior)
* [Examples](#examples)
* [Simple Pod Spec Example](#simple-pod-spec-example)
* [Pod Spec with `ConfigMap` Example](#pod-spec-with-`configmap`-example)
* [ReplicaSet with Pod Spec Example](#replicaset-with-pod-spec-example)
* [Multiple PodInjectionPolicy Example](#multiple-podinjectionpolicy-example)
* [Conflict Example](#conflict-example)
## Abstract
Describes a policy resource that allows for the loose coupling of a Pod's
definition from additional runtime requirements for that Pod. For example,
mounting of Secrets, or setting additional environment variables,
may not be known at Pod deployment time, but may be required at Pod creation
time.
## Motivation
Consuming a service involves more than just connectivity. In addition to
coordinates to reach the service, credentials and non-secret configuration
parameters are typically needed to use the service. The primitives for this
already exist, but a gap exists where loose coupling is desired: it should be
possible to inject pods with the information they need to use a service on a
service-by-service basis, without the pod authors having to incorporate the
information into every pod spec where it is needed.
## Constraints and Assumptions
1. Future work might require new mechanisms to be made to work with existing
controllers such as deployments and replicasets that create pods. Existing
controllers that create pods should recreate their pods when a new Pod Injection
Policy is added that would effect them.
## Use Cases
- As a user, I want to be able to provision a new pod
without needing to know the application configuration primitives the
services my pod will consume.
- As a cluster admin, I want specific configuration items of a service to be
withheld visibly from a developer deploying a service, but not to block the
developer from shipping.
- As an app developer, I want to provision a Cloud Spanner instance and then
access it from within my Kubernetes cluster.
- As an app developer, I want the Cloud Spanner provisioning process to
configure my Kubernetes cluster so the endpoints and credentials for my
Cloud Spanner instance are implicitly injected into Pods matching a label
selector (without me having to modify the PodSpec to add the specific
Configmap/Secret containing the endpoint/credential data).
**Specific Example:**
1. Database Administrator provisions a MySQL service for their cluster.
2. Database Administrator creates secrets for the cluster containing the
database name, username, and password.
3. Database Administrator creates a `PodInjectionPolicy` defining the database
port as an enviornment variable, as well as the secrets. See
[Examples](#examples) below for various examples.
4. Developer of an application can now label their pod with the specified
`Selector` the Database Administrator tells them, and consume the MySQL
database without needing to know any of the details from step 2 and 3.
### Summary
The use case we are targeting is to automatically inject into Pods the
information required to access non-Kubernetes-Services, such as accessing an
instances of Cloud Spanner. Accessing external services such as Cloud Spanner
may require the Pods to have specific credential and endpoint data.
Using a Pod Injection Policy allows pod template authors to not have to explicitly
set information for every pod. This way authors of pod templates consuming a
specific service do not need to know all the details about that service.
### Prior Art
Internally for Kubernetes we already support accessing the Kubernetes api from
all Pods by injecting the credentials and endpoint data automatically - e.g.
injecting the serviceaccount credentials into a volume (via secret) using an
[admission controller](https://github.com/kubernetes/kubernetes/blob/97212f5b3a2961d0b58a20bdb6bda3ccfa159bd7/plugin/pkg/admission/serviceaccount/admission.go),
and injecting the Service endpoints into environment
variables. This is done without the Pod explicitly mounting the serviceaccount
secret.
### Objectives
The goal of this proposal is to generalize these capabilities so we can introduce
similar support for accessing Services running external to the Kubernetes cluster.
We can assume that an appropriate Secret and Configmap have already been created
as part of the provisioning process of the external service. The need then is to
provide a mechanism for injecting the Secret and Configmap into Pods automatically.
The [ExplicitServiceLinks proposal](https://github.com/kubernetes/community/pull/176),
will allow us to decouple where a Service's credential and endpoint information
is stored in the Kubernetes cluster from a Pod's intent to access that Service
(e.g. in declaring it wants to access a Service, a Pod is automatically injected
with the credential and endpoint data required to do so).
## Proposed Changes
### PodInjectionPolicy API object
This resource is alpha. The policy itself is immutable. The API group will be
added to `apps` and the version is `v1alpha1`.
```go
// PodInjectionPolicy is a policy resource that defines additional runtime
// requirements for a Pod.
type PodInjectionPolicy struct {
unversioned.TypeMeta
ObjectMeta
// +optional
Spec PodInjectionPolicySpec
}
// PodInjectionPolicySpec is a description of a pod injection policy.
type PodInjectionPolicySpec struct {
// Selector is a label query over a set of resources, in this case pods.
// Required.
Selector unversioned.LabelSelector
// Env defines the collection of EnvVar to inject into containers.
// +optional
Env []EnvVar
// EnvFrom defines the collection of EnvFromSource to inject into
// containers.
// +optional
EnvFrom []EnvFromSource
// Volumes defines the collection of Volume to inject into the pod.
// +optional
Volumes []Volume `json:omitempty`
// VolumeMounts defines the collection of VolumeMount to inject into
// containers.
// +optional
VolumeMounts []VolumeMount
}
```
#### Validations
In order for the Pod Injection Policy to be valid it must fulfill the
following constraints:
- The `Selector` field must be defined. This is how we know which pods
to inject so therefore it is required and cannot be empty.
- The policy must define _at least_ 1 of `Env`, `EnvFrom`, or `Volumes` with
corresponding `VolumeMounts`.
- If you define a `Volume`, it has to define a `VolumeMount`.
- For `Env`, `EnvFrom`, `Volumes`, and `VolumeMounts` all existing API
validations are applied.
This resource will be immutable, if you want to change something you can delete
the old policy and recreate a new one. We can change this to be mutable in the
future but by disallowing it now, we will not break people in the future.
#### Conflicts
There are a number of edge conditions that might occur at the time of
injection. These are as follows:
- Merging lists with no conflicts: if a pod already has a `Volume`,
`VolumeMount` or `EnvVar` defined **exactly** as defined in the
PodInjectionPolicy. No error will occur since they are the exact same. The
motivation behind this is if services have no quite converted to using pod
injection policies yet and have duplicated information and an error should
obviously not be thrown if the items that need to be injected already exist
and are exactly the same.
- Merging lists with conflicts: if a PIP redefines an `EnvVar` or a `Volume`,
an event on the pod showing the error on the conflict will be thrown and
nothing will be injected.
- Conflicts between `Env` and `EnvFrom`: this would throw an error with an
event on the pod showing the error on the conflict. Nothing would be
injected.
> **Note:** In the case of a conflict nothing will be injected. The entire
> policy is ignored and an event is thrown on the pod detailing the conflict.
### AdmissionControl Plug-in: PodInjectionPolicy
The **PodInjectionPolicy** plug-in introspects all incoming pod creation
requests and injects the pod based off a `Selector` with the desired
attributes.
For the initial alpha, the order of precedence for applying multiple
`PodInjectionPolicy` specs is from oldest to newest. All Pod Injection
Policies in a namespace should be order agnostic; the order of application is
unspecified. Users should ensure that policies do not overlap.
However we can use merge keys to detect some of the conflicts that may occur.
This will not be enabled by default for all clusters, but once GA will be
a part of the set of strongly recommended plug-ins documented
[here](https://kubernetes.io/docs/admin/admission-controllers/#is-there-a-recommended-set-of-plug-ins-to-use).
**Why not an Initializer?**
This will be first implemented as an AdmissionControl plug-in then can be
converted to an Initializer once that is fully ready. The proposal for
Initializers can be found at [kubernetes/community#132](https://github.com/kubernetes/community/pull/132).
#### Behavior
This will modify the pod spec. The supported changes to
`Env`, `EnvFrom`, and `VolumeMounts` apply to the container spec for
all containers in the pod with the specified matching `Selector`. The
changes to `Volumes` apply to the pod spec for all pods matching `Selector`.
The resultant modified pod spec will be annotated to show that it was modified by
the `PodInjectionPolicy`. This will be of the form
`podinjectionpolicy.admission.kubernetes.io/<pip name>": "<resource version>"`.
*Why modify all containers in a pod?*
Currently there is no concept of labels on specific containers in a pod which
would be necessary for per-container pod injections. We could add labels
for specific containers which would allow this and be the best solution to not
injecting all. Container labels have been discussed various times through
multiple issues and proposals, which all congregate to this thread on the
[kubernetes-sig-node mailing
list](https://groups.google.com/forum/#!topic/kubernetes-sig-node/gijxbYC7HT8).
In the future, even if container labels were added, we would need to be careful
about not making breaking changes to the current behavior.
Other solutions include basing the container to inject based off
matching its name to another field in the `PodInjectionPolicy` spec, but
this would not scale well and would cause annoyance with configuration
management.
In the future we might question whether we need or want containers to express
that they expect injection. At this time we are deferring this issue.
## Examples
### Simple Pod Spec Example
This is a simple example to show how a Pod spec is modified by the Pod
Injection Policy.
**User submitted pod spec:**
```yaml
apiVersion: v1
kind: Pod
metadata:
name: website
labels:
app: website
role: frontend
spec:
containers:
- name: website
image: ecorp/website
ports:
- containerPort: 80
```
**Example Pod Injection Policy:**
```yaml
kind: PodInjectionPolicy
apiVersion: podinjection/v1alpha1
metadata:
name: allow-database
namespace: myns
spec:
selector:
matchLabels:
role: frontend
env:
- name: DB_PORT
value: 6379
volumeMounts:
- mountPath: /cache
name: cache-volume
volumes:
- name: cache-volume
emptyDir: {}
```
**Pod spec after admission controller:**
```yaml
apiVersion: v1
kind: Pod
metadata:
name: website
labels:
app: website
role: frontend
annotations:
podinjectionpolicy.admission.kubernetes.io/allow-database: "resource version"
spec:
containers:
- name: website
image: ecorp/website
volumeMounts:
- mountPath: /cache
name: cache-volume
ports:
- containerPort: 80
env:
- name: DB_PORT
value: 6379
volumes:
- name: cache-volume
emptyDir: {}
```
### Pod Spec with `ConfigMap` Example
This is an example to show how a Pod spec is modified by the Pod Injection
Policy that defines a `ConfigMap` for Environment Variables.
**User submitted pod spec:**
```yaml
apiVersion: v1
kind: Pod
metadata:
name: website
labels:
app: website
role: frontend
spec:
containers:
- name: website
image: ecorp/website
ports:
- containerPort: 80
```
**User submitted `ConfigMap`:**
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: etcd-env-config
data:
number_of_members: "1"
initial_cluster_state: new
initial_cluster_token: DUMMY_ETCD_INITIAL_CLUSTER_TOKEN
discovery_token: DUMMY_ETCD_DISCOVERY_TOKEN
discovery_url: http://etcd_discovery:2379
etcdctl_peers: http://etcd:2379
duplicate_key: FROM_CONFIG_MAP
REPLACE_ME: "a value"
```
**Example Pod Injection Policy:**
```yaml
kind: PodInjectionPolicy
apiVersion: podinjection/v1alpha1
metadata:
name: allow-database
namespace: myns
spec:
selector:
matchLabels:
role: frontend
env:
- name: DB_PORT
value: 6379
- name: duplicate_key
value: FROM_ENV
- name: expansion
value: $(REPLACE_ME)
envFrom:
- configMapRef:
name: etcd-env-config
volumeMounts:
- mountPath: /cache
name: cache-volume
- mountPath: /etc/app/config.json
readOnly: true
name: secret-volume
volumes:
- name: cache-volume
emptyDir: {}
- name: secret-volume
secretName: config-details
```
**Pod spec after admission controller:**
```yaml
apiVersion: v1
kind: Pod
metadata:
name: website
labels:
app: website
role: frontend
annotations:
podinjectionpolicy.admission.kubernetes.io/allow-database: "resource version"
spec:
containers:
- name: website
image: ecorp/website
volumeMounts:
- mountPath: /cache
name: cache-volume
- mountPath: /etc/app/config.json
readOnly: true
name: secret-volume
ports:
- containerPort: 80
env:
- name: DB_PORT
value: 6379
- name: duplicate_key
value: FROM_ENV
- name: expansion
value: $(REPLACE_ME)
envFrom:
- configMapRef:
name: etcd-env-config
volumes:
- name: cache-volume
emptyDir: {}
- name: secret-volume
secretName: config-details
```
### ReplicaSet with Pod Spec Example
The following example shows that only the pod spec is modified by the Pod
Injection Policy.
**User submitted ReplicaSet:**
```yaml
apiVersion: podinjection/v1alpha1
kind: ReplicaSet
metadata:
name: frontend
spec:
replicas: 3
selector:
matchLabels:
tier: frontend
matchExpressions:
- {key: tier, operator: In, values: [frontend]}
template:
metadata:
labels:
app: guestbook
tier: frontend
spec:
containers:
- name: php-redis
image: gcr.io/google_samples/gb-frontend:v3
resources:
requests:
cpu: 100m
memory: 100Mi
env:
- name: GET_HOSTS_FROM
value: dns
ports:
- containerPort: 80
```
**Example Pod Injection Policy:**
```yaml
kind: PodInjectionPolicy
apiVersion: podinjection/v1alpha1
metadata:
name: allow-database
namespace: myns
spec:
selector:
matchLabels:
tier: frontend
env:
- name: DB_PORT
value: 6379
volumeMounts:
- mountPath: /cache
name: cache-volume
volumes:
- name: cache-volume
emptyDir: {}
```
**Pod spec after admission controller:**
```yaml
kind: Pod
metadata:
labels:
app: guestbook
tier: frontend
annotations:
podinjectionpolicy.admission.kubernetes.io/allow-database: "resource version"
spec:
containers:
- name: php-redis
image: gcr.io/google_samples/gb-frontend:v3
resources:
requests:
cpu: 100m
memory: 100Mi
volumeMounts:
- mountPath: /cache
name: cache-volume
env:
- name: GET_HOSTS_FROM
value: dns
- name: DB_PORT
value: 6379
ports:
- containerPort: 80
volumes:
- name: cache-volume
emptyDir: {}
```
### Multiple PodInjectionPolicy Example
This is an example to show how a Pod spec is modified by multiple Pod
Injection Policies.
**User submitted pod spec:**
```yaml
apiVersion: v1
kind: Pod
metadata:
name: website
labels:
app: website
role: frontend
spec:
containers:
- name: website
image: ecorp/website
ports:
- containerPort: 80
```
**Example Pod Injection Policy:**
```yaml
kind: PodInjectionPolicy
apiVersion: podinjection/v1alpha1
metadata:
name: allow-database
namespace: myns
spec:
selector:
matchLabels:
role: frontend
env:
- name: DB_PORT
value: 6379
volumeMounts:
- mountPath: /cache
name: cache-volume
volumes:
- name: cache-volume
emptyDir: {}
```
**Another Pod Injection Policy:**
```yaml
kind: PodInjectionPolicy
apiVersion: podinjection/v1alpha1
metadata:
name: proxy
namespace: myns
spec:
selector:
matchLabels:
role: frontend
volumeMounts:
- mountPath: /etc/proxy/configs
name: proxy-volume
volumes:
- name: proxy-volume
emptyDir: {}
```
**Pod spec after admission controller:**
```yaml
apiVersion: v1
kind: Pod
metadata:
name: website
labels:
app: website
role: frontend
annotations:
podinjectionpolicy.admission.kubernetes.io/allow-database: "resource version"
podinjectionpolicy.admission.kubernetes.io/proxy: "resource version"
spec:
containers:
- name: website
image: ecorp/website
volumeMounts:
- mountPath: /cache
name: cache-volume
- mountPath: /etc/proxy/configs
name: proxy-volume
ports:
- containerPort: 80
env:
- name: DB_PORT
value: 6379
volumes:
- name: cache-volume
emptyDir: {}
- name: proxy-volume
emptyDir: {}
```
### Conflict Example
This is a example to show how a Pod spec is not modified by the Pod Injection
Policy when there is a conflict.
**User submitted pod spec:**
```yaml
apiVersion: v1
kind: Pod
metadata:
name: website
labels:
app: website
role: frontend
spec:
containers:
- name: website
image: ecorp/website
volumeMounts:
- mountPath: /cache
name: cache-volume
ports:
volumes:
- name: cache-volume
emptyDir: {}
- containerPort: 80
```
**Example Pod Injection Policy:**
```yaml
kind: PodInjectionPolicy
apiVersion: podinjection/v1alpha1
metadata:
name: allow-database
namespace: myns
spec:
selector:
matchLabels:
role: frontend
env:
- name: DB_PORT
value: 6379
volumeMounts:
- mountPath: /cache
name: other-volume
volumes:
- name: other-volume
emptyDir: {}
```
**Pod spec after admission controller will not change because of the conflict:**
```yaml
apiVersion: v1
kind: Pod
metadata:
name: website
labels:
app: website
role: frontend
spec:
containers:
- name: website
image: ecorp/website
volumeMounts:
- mountPath: /cache
name: cache-volume
ports:
volumes:
- name: cache-volume
emptyDir: {}
- containerPort: 80
```
**If we run `kubectl describe...` we can see the event:**
```
$ kubectl describe ...
....
Events:
FirstSeen LastSeen Count From SubobjectPath Reason Message
Tue, 07 Feb 2017 16:56:12 -0700 Tue, 07 Feb 2017 16:56:12 -0700 1 {podinjectionpolicy.admission.kubernetes.io/allow-database } conflict Conflict on pod injection policy. Duplicate mountPath /cache.
```