Added final reports
This commit is contained in:
Aaron Small
parent
a4ead5f14d
commit
a24e8c3c93
|
|
@ -29,9 +29,30 @@ Perform a security audit on k8s with a vendor and produce as artifacts a threat
|
|||
* [Open Community Issues/PRs](https://github.com/kubernetes/community/labels/wg%2Fsecurity-audit)
|
||||
|
||||
<!-- BEGIN CUSTOM CONTENT -->
|
||||
## Published Documents
|
||||
|
||||
Trail of Bits and Atredis Partners, in collaboration with the Security Audit Working Group, have released the following documents which
|
||||
detail their assessment of Kubernetes security posture and their findings.
|
||||
|
||||
### Findings
|
||||
|
||||
* [Kubernetes Security Review](findings/Kubernetes%20Final%20Report.pdf)
|
||||
* [Attacking and Defending Kubernetes Installations](findings/AtredisPartners_Attacking_Kubernetes-v1.0.pdf)
|
||||
* [Whitepaper](findings/Kubernetes%20White%20Paper.pdf)
|
||||
* [Threat Model](findings/Kubernetes%20Threat%20Model.pdf)
|
||||
|
||||
### Ancillary Data
|
||||
|
||||
* [Rapid Risk Assessments](ancillary-data/rapid-risk-assessments)
|
||||
* [Dataflow](ancillary-data/dataflow)
|
||||
|
||||
## Mailing Lists
|
||||
|
||||
* Sensitive communications regarding the audit should be sent to the [private variant of the mailing list](https://groups.google.com/forum/#!forum/kubernetes-wg-security-audit-private).
|
||||
|
||||
## Request For Proposals
|
||||
|
||||
The RFP will be open between 2018/10/29 and 2018/11/30 and has been published [here](https://github.com/kubernetes/community/blob/master/wg-security-audit/RFP.md).
|
||||
The RFP was open between 2018/10/29 and 2018/11/30 and has been published [here](https://github.com/kubernetes/community/blob/master/wg-security-audit/RFP.md).
|
||||
|
||||
## Vendor Selection
|
||||
|
||||
|
|
@ -39,8 +60,4 @@ The [RFP](https://github.com/kubernetes/community/blob/master/wg-security-audit/
|
|||
|
||||
You can read more about the vendor selection [here](RFP_Decision.md).
|
||||
|
||||
## Mailing Lists
|
||||
|
||||
* Sensitive communications regarding the audit should be sent to the [private variant of the mailing list](https://groups.google.com/forum/#!forum/kubernetes-wg-security-audit-private).
|
||||
|
||||
<!-- END CUSTOM CONTENT -->
|
||||
|
|
|
|||
|
|
@ -0,0 +1,47 @@
|
|||
digraph K8S {
|
||||
subgraph cluster_apiserverinternal {
|
||||
node [style=filled];
|
||||
color=green;
|
||||
etcd[label="etcd"];
|
||||
label = "API Server Data Layer";
|
||||
}
|
||||
|
||||
subgraph cluster_apiserver {
|
||||
node [style=filled];
|
||||
color=blue;
|
||||
kubeapiserver[label="kube-apiserver"];
|
||||
kubeapiserver->etcd[label="HTTPS"]
|
||||
label = "API Server";
|
||||
}
|
||||
|
||||
subgraph cluster_mastercomponents {
|
||||
node [style=filled];
|
||||
label = "Master Control Plane Components";
|
||||
scheduler[label="Scheduler"];
|
||||
controllers[label="Controllers"]
|
||||
scheduler->kubeapiserver[label="Callback/HTTPS"];
|
||||
controllers->kubeapiserver[label="Callback/HTTPS"];
|
||||
color=black;
|
||||
}
|
||||
|
||||
subgraph cluster_worker {
|
||||
label="Worker"
|
||||
color="blue"
|
||||
kubelet->kubeapiserver[label="authenticated HTTPS"]
|
||||
kubeproxy[label="kube-proxy"]
|
||||
iptables->kubeproxy->iptables
|
||||
pods[label="pods with various containers"]
|
||||
pods->kubeproxy->pods
|
||||
}
|
||||
|
||||
subgraph cluster_internet {
|
||||
label="Internet"
|
||||
authuser[label="Authorized User via kubebctl"]
|
||||
generaluser[label="General User"]
|
||||
authuser->kubeapiserver[label="Authenticated HTTPS"]
|
||||
generaluser->pods[label="application-specific connection protocol"]
|
||||
}
|
||||
kubeapiserver->kubelet[label="HTTPS"]
|
||||
kubeapiserver->pods[label="HTTP",color=red]
|
||||
}
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 100 KiB |
|
|
@ -0,0 +1,3 @@
|
|||
python3 tm.py --dfd > updated-dataflow.dot
|
||||
dot -Tpng < updated-dataflow.dot > updated-dataflow.png
|
||||
open updated-dataflow.png
|
||||
|
|
@ -0,0 +1 @@
|
|||
pytm==0.4
|
||||
|
|
@ -0,0 +1,106 @@
|
|||
# !/usr/bin/env python3
|
||||
|
||||
from pytm.pytm import TM, Server, Datastore, Dataflow, Boundary, Actor, Lambda, Process
|
||||
|
||||
tm = TM("Kubernetes Threat Model")
|
||||
tm.description = "a deep-dive threat model of Kubernetes"
|
||||
|
||||
# Boundaries
|
||||
|
||||
inet = Boundary("Internet")
|
||||
mcdata = Boundary("Master Control Data")
|
||||
apisrv = Boundary("API Server")
|
||||
mcomps = Boundary("Master Control Components")
|
||||
worker = Boundary("Worker")
|
||||
contain = Boundary("Container")
|
||||
|
||||
# Actors
|
||||
|
||||
miu = Actor("Malicious Internal User")
|
||||
ia = Actor("Internal Attacker")
|
||||
ea = Actor("External Actor")
|
||||
admin = Actor("Administrator")
|
||||
dev = Actor("Developer")
|
||||
eu = Actor("End User")
|
||||
|
||||
# Server & OS Components
|
||||
|
||||
etcd = Datastore("N-ary etcd servers")
|
||||
apiserver = Server("kube-apiserver")
|
||||
kubelet = Server("kubelet")
|
||||
kubeproxy = Server("kube-proxy")
|
||||
scheduler = Server("kube-scheduler")
|
||||
controllers = Server("CCM/KCM")
|
||||
pods = Server("Pods")
|
||||
iptables = Process("iptables")
|
||||
|
||||
# Component <> Boundary Relations
|
||||
etcd.inBoundary = mcdata
|
||||
mcdata.inBoundary = apisrv
|
||||
apiserver.inBoundary = apisrv
|
||||
kubelet.inBoundary = worker
|
||||
kubeproxy.inBoundary = worker
|
||||
pods.inBoundary = contain
|
||||
scheduler.inBoundary = mcomps
|
||||
controllers.inBoundary = mcomps
|
||||
pods.inBoundary = contain
|
||||
iptables.inBoundary = worker
|
||||
miu.inBoundary = apisrv
|
||||
ia.inBoundary = contain
|
||||
ea.inBoundary = inet
|
||||
admin.inBoundary = apisrv
|
||||
dev.inBoundary = inet
|
||||
eu.inBoundary = inet
|
||||
|
||||
# Dataflows
|
||||
|
||||
apiserver2etcd = Dataflow(apiserver, etcd, "All kube-apiserver data")
|
||||
apiserver2etcd.isEncrypted = True
|
||||
apiserver2etcd.protocol = "HTTPS"
|
||||
|
||||
apiserver2kubelet = Dataflow(apiserver, kubelet, "kubelet Health, Status, &c.")
|
||||
apiserver2kubelet.isEncrypted = False
|
||||
apiserver2kubelet.protocol = "HTTP"
|
||||
|
||||
apiserver2kubeproxy = Dataflow(apiserver, kubeproxy, "kube-proxy Health, Status, &c.")
|
||||
apiserver2kubeproxy.isEncrypted = False
|
||||
apiserver2kubeproxy.protocol = "HTTP"
|
||||
|
||||
apiserver2scheduler = Dataflow(apiserver, scheduler, "kube-scheduler Health, Status, &c.")
|
||||
apiserver2scheduler.isEncrypted = False
|
||||
apiserver2scheduler.protocol = "HTTP"
|
||||
|
||||
apiserver2controllers = Dataflow(apiserver, controllers, "{kube, cloud}-controller-manager Health, Status, &c.")
|
||||
apiserver2controllers.isEncrypted = False
|
||||
apiserver2controllers.protocol = "HTTP"
|
||||
|
||||
kubelet2apiserver = Dataflow(kubelet, apiserver, "HTTP watch for resources on kube-apiserver")
|
||||
kubelet2apiserver.isEncrypted = True
|
||||
kubelet2apiserver.protocol = "HTTPS"
|
||||
|
||||
kubeproxy2apiserver = Dataflow(kubeproxy, apiserver, "HTTP watch for resources on kube-apiserver")
|
||||
kubeproxy2apiserver.isEncrypted = True
|
||||
kubeproxy2apiserver.protocol = "HTTPS"
|
||||
|
||||
controllers2apiserver = Dataflow(controllers, apiserver, "HTTP watch for resources on kube-apiserver")
|
||||
controllers2apiserver.isEncrypted = True
|
||||
controllers2apiserver.protocol = "HTTPS"
|
||||
|
||||
scheduler2apiserver = Dataflow(scheduler, apiserver, "HTTP watch for resources on kube-apiserver")
|
||||
scheduler2apiserver.isEncrypted = True
|
||||
scheduler2apiserver.protocol = "HTTPS"
|
||||
|
||||
kubelet2iptables = Dataflow(kubelet, iptables, "kubenet update of iptables (... ipvs, &c) to setup Host-level ports")
|
||||
kubelet2iptables.protocol = "IPC"
|
||||
|
||||
kubeproxy2iptables = Dataflow(kubeproxy, iptables, "kube-prxy update of iptables (... ipvs, &c) to setup all pod networking")
|
||||
kubeproxy2iptables.protocol = "IPC"
|
||||
|
||||
kubelet2pods = Dataflow(kubelet, pods, "kubelet to pod/CRI runtime, to spin up pods within a host")
|
||||
kubelet2pods.protocol = "IPC"
|
||||
|
||||
eu2pods = Dataflow(eu, pods, "End-user access of Kubernetes-hosted applications")
|
||||
ea2pods = Dataflow(ea, pods, "External Attacker attempting to compromise a trust boundary")
|
||||
ia2cnts = Dataflow(ia, pods, "Internal Attacker with access to a compromised or malicious pod")
|
||||
|
||||
tm.process()
|
||||
|
|
@ -0,0 +1,217 @@
|
|||
digraph tm {
|
||||
graph [
|
||||
fontname = Arial;
|
||||
fontsize = 14;
|
||||
]
|
||||
node [
|
||||
fontname = Arial;
|
||||
fontsize = 14;
|
||||
rankdir = lr;
|
||||
]
|
||||
edge [
|
||||
shape = none;
|
||||
fontname = Arial;
|
||||
fontsize = 12;
|
||||
]
|
||||
labelloc = "t";
|
||||
fontsize = 20;
|
||||
nodesep = 1;
|
||||
|
||||
subgraph cluster_bfaefefcfbeeafeefac {
|
||||
graph [
|
||||
fontsize = 10;
|
||||
fontcolor = firebrick2;
|
||||
style = dashed;
|
||||
color = firebrick2;
|
||||
label = <<i>Internet</i>>;
|
||||
]
|
||||
|
||||
bfbeacdafaceebdccfdffcdfcedfec [
|
||||
shape = square;
|
||||
label = <<table border="0" cellborder="0" cellpadding="2"><tr><td><b>External Actor</b></td></tr></table>>;
|
||||
]
|
||||
abaadcacbbafdffbcffffbeedef [
|
||||
shape = square;
|
||||
label = <<table border="0" cellborder="0" cellpadding="2"><tr><td><b>Developer</b></td></tr></table>>;
|
||||
]
|
||||
adafdaeaedeedcafe [
|
||||
shape = square;
|
||||
label = <<table border="0" cellborder="0" cellpadding="2"><tr><td><b>End User</b></td></tr></table>>;
|
||||
]
|
||||
|
||||
}
|
||||
|
||||
subgraph cluster_bbfdadaacbdaedcebfec {
|
||||
graph [
|
||||
fontsize = 10;
|
||||
fontcolor = firebrick2;
|
||||
style = dashed;
|
||||
color = firebrick2;
|
||||
label = <<i>Master Control Data</i>>;
|
||||
]
|
||||
|
||||
bfffcaeeeeedccabfaaeff [
|
||||
shape = none;
|
||||
color = black;
|
||||
label = <<table sides="TB" cellborder="0" cellpadding="2"><tr><td><font color="black"><b>N-ary etcd servers</b></font></td></tr></table>>;
|
||||
]
|
||||
|
||||
}
|
||||
|
||||
subgraph cluster_afeffbbfdbeeefcabddacdba {
|
||||
graph [
|
||||
fontsize = 10;
|
||||
fontcolor = firebrick2;
|
||||
style = dashed;
|
||||
color = firebrick2;
|
||||
label = <<i>API Server</i>>;
|
||||
]
|
||||
|
||||
bdfbefabdbefeacdfcabaac [
|
||||
shape = square;
|
||||
label = <<table border="0" cellborder="0" cellpadding="2"><tr><td><b>Malicious Internal User</b></td></tr></table>>;
|
||||
]
|
||||
fabeebdadbcdffdcdec [
|
||||
shape = square;
|
||||
label = <<table border="0" cellborder="0" cellpadding="2"><tr><td><b>Administrator</b></td></tr></table>>;
|
||||
]
|
||||
eadddadcfbabebaed [
|
||||
shape = circle
|
||||
color = black
|
||||
label = <<table border="0" cellborder="0" cellpadding="2"><tr><td><b>kube-apiserver</b></td></tr></table>>;
|
||||
]
|
||||
|
||||
}
|
||||
|
||||
subgraph cluster_cebcbebffccbfedcaffbb {
|
||||
graph [
|
||||
fontsize = 10;
|
||||
fontcolor = firebrick2;
|
||||
style = dashed;
|
||||
color = firebrick2;
|
||||
label = <<i>Master Control Components</i>>;
|
||||
]
|
||||
|
||||
ffceacecdbcacdddddffbfa [
|
||||
shape = circle
|
||||
color = black
|
||||
label = <<table border="0" cellborder="0" cellpadding="2"><tr><td><b>kube-scheduler</b></td></tr></table>>;
|
||||
]
|
||||
adffdceecfcfbcfdaefca [
|
||||
shape = circle
|
||||
color = black
|
||||
label = <<table border="0" cellborder="0" cellpadding="2"><tr><td><b>CCM/KCM</b></td></tr></table>>;
|
||||
]
|
||||
|
||||
}
|
||||
|
||||
subgraph cluster_baaffdafbdceebaaafaefeea {
|
||||
graph [
|
||||
fontsize = 10;
|
||||
fontcolor = firebrick2;
|
||||
style = dashed;
|
||||
color = firebrick2;
|
||||
label = <<i>Worker</i>>;
|
||||
]
|
||||
|
||||
dbddcfaeaacebaecba [
|
||||
shape = circle
|
||||
color = black
|
||||
label = <<table border="0" cellborder="0" cellpadding="2"><tr><td><b>kubelet</b></td></tr></table>>;
|
||||
]
|
||||
ddcaffdfdebdaeff [
|
||||
shape = circle
|
||||
color = black
|
||||
label = <<table border="0" cellborder="0" cellpadding="2"><tr><td><b>kube-proxy</b></td></tr></table>>;
|
||||
]
|
||||
bcdcebabbdaadffeaeddcce [
|
||||
shape = circle;
|
||||
color = black;
|
||||
|
||||
label = <<table border="0" cellborder="0" cellpadding="2"><tr><td><font color="black"><b>iptables</b></font></td></tr></table>>;
|
||||
]
|
||||
|
||||
}
|
||||
|
||||
subgraph cluster_fdcecbcfbeadaccab {
|
||||
graph [
|
||||
fontsize = 10;
|
||||
fontcolor = firebrick2;
|
||||
style = dashed;
|
||||
color = firebrick2;
|
||||
label = <<i>Container</i>>;
|
||||
]
|
||||
|
||||
bdfadfbeeaedceab [
|
||||
shape = square;
|
||||
label = <<table border="0" cellborder="0" cellpadding="2"><tr><td><b>Internal Attacker</b></td></tr></table>>;
|
||||
]
|
||||
eefbffbeaaeecaceaaabe [
|
||||
shape = circle
|
||||
color = black
|
||||
label = <<table border="0" cellborder="0" cellpadding="2"><tr><td><b>Pods</b></td></tr></table>>;
|
||||
]
|
||||
|
||||
}
|
||||
|
||||
eadddadcfbabebaed -> bfffcaeeeeedccabfaaeff [
|
||||
color = black;
|
||||
label = <<table border="0" cellborder="0" cellpadding="2"><tr><td><font color ="black"><b>All kube-apiserver data</b></font></td></tr></table>>;
|
||||
]
|
||||
eadddadcfbabebaed -> dbddcfaeaacebaecba [
|
||||
color = black;
|
||||
label = <<table border="0" cellborder="0" cellpadding="2"><tr><td><font color ="black"><b>kubelet Health, Status, &c.</b></font></td></tr></table>>;
|
||||
]
|
||||
eadddadcfbabebaed -> ddcaffdfdebdaeff [
|
||||
color = black;
|
||||
label = <<table border="0" cellborder="0" cellpadding="2"><tr><td><font color ="black"><b>kube-proxy Health, Status, &c.</b></font></td></tr></table>>;
|
||||
]
|
||||
eadddadcfbabebaed -> ffceacecdbcacdddddffbfa [
|
||||
color = black;
|
||||
label = <<table border="0" cellborder="0" cellpadding="2"><tr><td><font color ="black"><b>kube-scheduler Health, Status, &c.</b></font></td></tr></table>>;
|
||||
]
|
||||
eadddadcfbabebaed -> adffdceecfcfbcfdaefca [
|
||||
color = black;
|
||||
label = <<table border="0" cellborder="0" cellpadding="2"><tr><td><font color ="black"><b>{kube, cloud}-controller-manager Health, Status, &c.</b></font></td></tr></table>>;
|
||||
]
|
||||
dbddcfaeaacebaecba -> eadddadcfbabebaed [
|
||||
color = black;
|
||||
label = <<table border="0" cellborder="0" cellpadding="2"><tr><td><font color ="black"><b>HTTP watch for resources on kube-apiserver</b></font></td></tr></table>>;
|
||||
]
|
||||
ddcaffdfdebdaeff -> eadddadcfbabebaed [
|
||||
color = black;
|
||||
label = <<table border="0" cellborder="0" cellpadding="2"><tr><td><font color ="black"><b>HTTP watch for resources on kube-apiserver</b></font></td></tr></table>>;
|
||||
]
|
||||
adffdceecfcfbcfdaefca -> eadddadcfbabebaed [
|
||||
color = black;
|
||||
label = <<table border="0" cellborder="0" cellpadding="2"><tr><td><font color ="black"><b>HTTP watch for resources on kube-apiserver</b></font></td></tr></table>>;
|
||||
]
|
||||
ffceacecdbcacdddddffbfa -> eadddadcfbabebaed [
|
||||
color = black;
|
||||
label = <<table border="0" cellborder="0" cellpadding="2"><tr><td><font color ="black"><b>HTTP watch for resources on kube-apiserver</b></font></td></tr></table>>;
|
||||
]
|
||||
dbddcfaeaacebaecba -> bcdcebabbdaadffeaeddcce [
|
||||
color = black;
|
||||
label = <<table border="0" cellborder="0" cellpadding="2"><tr><td><font color ="black"><b>kubenet update of iptables (... ipvs, &c) to setup Host-level ports</b></font></td></tr></table>>;
|
||||
]
|
||||
ddcaffdfdebdaeff -> bcdcebabbdaadffeaeddcce [
|
||||
color = black;
|
||||
label = <<table border="0" cellborder="0" cellpadding="2"><tr><td><font color ="black"><b>kube-prxy update of iptables (... ipvs, &c) to setup all pod networking</b></font></td></tr></table>>;
|
||||
]
|
||||
dbddcfaeaacebaecba -> eefbffbeaaeecaceaaabe [
|
||||
color = black;
|
||||
label = <<table border="0" cellborder="0" cellpadding="2"><tr><td><font color ="black"><b>kubelet to pod/CRI runtime, to spin up pods within a host</b></font></td></tr></table>>;
|
||||
]
|
||||
adafdaeaedeedcafe -> eefbffbeaaeecaceaaabe [
|
||||
color = black;
|
||||
label = <<table border="0" cellborder="0" cellpadding="2"><tr><td><font color ="black"><b>End-user access of Kubernetes-hosted applications</b></font></td></tr></table>>;
|
||||
]
|
||||
bfbeacdafaceebdccfdffcdfcedfec -> eefbffbeaaeecaceaaabe [
|
||||
color = black;
|
||||
label = <<table border="0" cellborder="0" cellpadding="2"><tr><td><font color ="black"><b>External Attacker attempting to compromise a trust boundary</b></font></td></tr></table>>;
|
||||
]
|
||||
bdfadfbeeaedceab -> eefbffbeaaeecaceaaabe [
|
||||
color = black;
|
||||
label = <<table border="0" cellborder="0" cellpadding="2"><tr><td><font color ="black"><b>Internal Attacker with access to a compromised or malicious pod</b></font></td></tr></table>>;
|
||||
]
|
||||
}
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 314 KiB |
|
|
@ -0,0 +1,141 @@
|
|||
# Overview
|
||||
|
||||
- Component: Container Runtime
|
||||
- Owner(s): [sig-node](https://github.com/kubernetes/community/blob/master/sig-node/README.md)
|
||||
- SIG/WG(s) at meeting:
|
||||
- Service Data Classification: High
|
||||
- Highest Risk Impact:
|
||||
|
||||
# Service Notes
|
||||
|
||||
The portion should walk through the component and discuss connections, their relevant controls, and generally lay out how the component serves its relevant function. For example
|
||||
a component that accepts an HTTP connection may have relevant questions about channel security (TLS and Cryptography), authentication, authorization, non-repudiation/auditing,
|
||||
and logging. The questions aren't the *only* drivers as to what may be spoken about, the questions are meant to drive what we discuss and keep things on task for the duration
|
||||
of a meeting/call.
|
||||
|
||||
## How does the service work?
|
||||
|
||||
- Container Runtimes expose an IPC endpoint such as a file system socket
|
||||
- kubelet retrieves pods to be executed from the kube-apiserver
|
||||
- The Container Runtime Interface then executes the necessary commands/requests from the actual container system (e.g. docker) to run the pod
|
||||
|
||||
## Are there any subcomponents or shared boundaries?
|
||||
|
||||
Yes
|
||||
|
||||
- The Container Runtime technically interfaces with kublet, and runs on the same host
|
||||
- However, the Container Runtime is logically a separate Trust Zone within the node
|
||||
|
||||
## What communications protocols does it use?
|
||||
|
||||
Various, depends on the IPC mechanism required by the Container Runtime
|
||||
|
||||
## Where does it store data?
|
||||
|
||||
Most data should be provided by kubelet or the CRI in running the container
|
||||
|
||||
## What is the most sensitive data it stores?
|
||||
|
||||
N/A
|
||||
|
||||
## How is that data stored?
|
||||
|
||||
N/A
|
||||
|
||||
# Meeting Notes
|
||||
|
||||
|
||||
# Data Dictionary
|
||||
|
||||
| Name | Classification/Sensitivity | Comments |
|
||||
| :--: | :--: | :--: |
|
||||
| Data | Goes | Here |
|
||||
|
||||
# Control Families
|
||||
|
||||
These are the areas of controls that we're interested in based on what the audit working group selected.
|
||||
|
||||
When we say "controls," we mean a logical section of an application or system that handles a security requirement. Per CNSSI:
|
||||
|
||||
> The management, operational, and technical controls (i.e., safeguards or countermeasures) prescribed for an information system to protect the confidentiality, integrity, and availability of the system and its information.
|
||||
|
||||
For example, an system may have authorization requirements that say:
|
||||
|
||||
- users must be registered with a central authority
|
||||
- all requests must be verified to be owned by the requesting user
|
||||
- each account must have attributes associated with it to uniquely identify the user
|
||||
|
||||
and so on.
|
||||
|
||||
For this assessment, we're looking at six basic control families:
|
||||
|
||||
- Networking
|
||||
- Cryptography
|
||||
- Secrets Management
|
||||
- Authentication
|
||||
- Authorization (Access Control)
|
||||
- Multi-tenancy Isolation
|
||||
|
||||
Obviously we can skip control families as "not applicable" in the event that the component does not require it. For example,
|
||||
something with the sole purpose of interacting with the local file system may have no meaningful Networking component; this
|
||||
isn't a weakness, it's simply "not applicable."
|
||||
|
||||
For each control family we want to ask:
|
||||
|
||||
- What does the component do for this control?
|
||||
- What sorts of data passes through that control?
|
||||
- for example, a component may have sensitive data (Secrets Management), but that data never leaves the component's storage via Networking
|
||||
- What can attacker do with access to this component?
|
||||
- What's the simplest attack against it?
|
||||
- Are there mitigations that we recommend (i.e. "Always use an interstitial firewall")?
|
||||
- What happens if the component stops working (via DoS or other means)?
|
||||
- Have there been similar vulnerabilities in the past? What were the mitigations?
|
||||
|
||||
# Threat Scenarios
|
||||
|
||||
- An External Attacker without access to the client application
|
||||
- An External Attacker with valid access to the client application
|
||||
- An Internal Attacker with access to cluster
|
||||
- A Malicious Internal User
|
||||
|
||||
## Networking
|
||||
|
||||
- CRI Runs an HTTP server
|
||||
- port forwarding, exec, attach
|
||||
- !FINDING TLS bye default, but not mutual TLS, and self-signed
|
||||
- kubelet -> exec request to CRI over gRPC
|
||||
- Returns URL with single use Token
|
||||
- gRPC is Unix Domain by default
|
||||
- Kubelet proxies or responds w/ redirect to API server (locally hosted CRI only)
|
||||
- !FINDING(same HTTP finding for pull as kubectl) CRI actually pulls images, no egress filtering
|
||||
- image tag is SHA256, CRI checks that
|
||||
- Not sure how CNI, it might be exec
|
||||
- only responds to connections
|
||||
- CRI uses Standard Go HTTP
|
||||
|
||||
## Cryptography
|
||||
|
||||
- Nothing beyond TLS
|
||||
|
||||
## Secrets Management
|
||||
|
||||
- !FINDING auth'd container repos, passed in via podspec, fetched by kubelet, are passed via CLI
|
||||
- so anyone with access to the host running the container can see those secrets
|
||||
|
||||
## Authentication
|
||||
|
||||
- Unix Domain Socket for gRPC, so Linux authN/authZ
|
||||
- !FINDING 8 character random single use token with 1 minute lifetype (response to line 109)
|
||||
|
||||
## Authorization
|
||||
|
||||
- no authZ
|
||||
|
||||
## Multi-tenancy Isolation
|
||||
|
||||
- knows nothing about tenants or namespaces
|
||||
- low-level component, kubelet/api-server is the arbiter
|
||||
|
||||
## Summary
|
||||
|
||||
# Recommendations
|
||||
|
|
@ -0,0 +1,162 @@
|
|||
# Overview
|
||||
|
||||
- Component: etcd
|
||||
- Owner(s): Technically external to Kubernetes itself, but managed by [sig-api-machinery](https://github.com/kubernetes/community/tree/master/sig-api-machinery)
|
||||
- SIG/WG(s) at meeting:
|
||||
- Service Data Classification: Critical (on a cluster with an API server, access to etcd is root access to the cluster)
|
||||
- Highest Risk Impact:
|
||||
|
||||
# Service Notes
|
||||
|
||||
The portion should walk through the component and discuss connections, their relevant controls, and generally lay out how the component serves its relevant function. For example
|
||||
a component that accepts an HTTP connection may have relevant questions about channel security (TLS and Cryptography), authentication, authorization, non-repudiation/auditing,
|
||||
and logging. The questions aren't the *only* drivers as to what may be spoken about, the questions are meant to drive what we discuss and keep things on task for the duration
|
||||
of a meeting/call.
|
||||
|
||||
## How does the service work?
|
||||
|
||||
- Distributed key-value store
|
||||
- uses RAFT for consensus
|
||||
- always need to deploy (N x M) + 1 members to avoid leader election issues
|
||||
- five is recommended for production usage
|
||||
- listens for requests from clients
|
||||
- clients are simple REST clients that interact via JSON or other mechanisms
|
||||
- in Kubernetes' case, data is stored under `/registry`
|
||||
|
||||
## Are there any subcomponents or shared boundaries?
|
||||
|
||||
There shouldn't be; documentation specifically states:
|
||||
|
||||
- should be in own cluster
|
||||
- limited to access by the API server(s) only
|
||||
- should use some sort of authentication (hopefully certificate auth)
|
||||
|
||||
## What communications protocols does it use?
|
||||
|
||||
- HTTPS (with optional client-side or two-way TLS)
|
||||
- can also use basic auth
|
||||
- there's technically gRPC as well
|
||||
|
||||
## Where does it store data?
|
||||
|
||||
- typical database-style:
|
||||
- data directory
|
||||
- snapshot directory
|
||||
- write-ahead log (WAL) directory
|
||||
- all three may be the same, depends on command line options
|
||||
- Consensus is then achieved across nodes via RAFT (leader election + log replication via distributed state machine)
|
||||
|
||||
## What is the most sensitive data it stores?
|
||||
|
||||
- literally holds the keys to the kingdom:
|
||||
- pod specs
|
||||
- secrets
|
||||
- roles/attributes for {R, A}BAC
|
||||
- literally any data stored in Kubernetes via the kube-apiserver
|
||||
- [Access to etcd is equivalent to root permission in the cluster](https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/#securing-etcd-clusters)
|
||||
|
||||
## How is that data stored?
|
||||
|
||||
- Outside the scope of this assessment per se, but not encrypted at rest
|
||||
- Kubernetes supports this itself with Encryption providers
|
||||
- the typical process of a WAL + data + snapshot is used
|
||||
- this is then replicated across the cluster with Raft
|
||||
|
||||
# Meeting Notes
|
||||
|
||||
- No authorization (from k8s perspective)
|
||||
- AUthentication by local port access in current k8s
|
||||
- working towards mTLS for all connections
|
||||
- Raft consensus port, listener port
|
||||
- backups in etcd (system-level) not encrypted
|
||||
- metrics aren't encrypted at all either
|
||||
- multi-tenant: no multi-tenant controls at all
|
||||
- the kube-apiserver is the arbiter namespaces
|
||||
- could add namespaces to the registry, but that is a large amount of work
|
||||
- no migration plan or test
|
||||
- watches (like kubelet watching for pod spec changes) would break
|
||||
- multi-single tenant is best route
|
||||
- RAFT port may be open by default, even in single etcd configuraitons
|
||||
- runs in a container within static Master kubelet, but is run as root
|
||||
- [CONTROL WEAKNESS] CA is passed on command line
|
||||
- Types of files: WAL, Snapshot, Data file (and maybe backup)
|
||||
- [FINDING] no checksums on WAL/Snapshot/Data
|
||||
- [RECOMMENDATION] checksum individual WAL entries, checksum the entire snapshot file
|
||||
- do this because it's fast enough for individual entries, and then the snapshot should never change
|
||||
- Crypto, really only TLS (std go) and checksums for backups (but not other files, as noted above)
|
||||
- No auditing, but that's less useful
|
||||
- kube-apiserver is the arbiter of what things are
|
||||
- kube-apiserver uses a single connection credential to etcd w/o impersonation, so harder to tell who did what
|
||||
- major events end up in the app log
|
||||
- debug mode allows you to see all events when they happen
|
||||
|
||||
# Data Dictionary
|
||||
|
||||
| Name | Classification/Sensitivity | Comments |
|
||||
| :--: | :--: | :--: |
|
||||
| Data | Goes | Here |
|
||||
|
||||
# Control Families
|
||||
|
||||
These are the areas of controls that we're interested in based on what the audit working group selected.
|
||||
|
||||
When we say "controls," we mean a logical section of an application or system that handles a security requirement. Per CNSSI:
|
||||
|
||||
> The management, operational, and technical controls (i.e., safeguards or countermeasures) prescribed for an information system to protect the confidentiality, integrity, and availability of the system and its information.
|
||||
|
||||
For example, an system may have authorization requirements that say:
|
||||
|
||||
- users must be registered with a central authority
|
||||
- all requests must be verified to be owned by the requesting user
|
||||
- each account must have attributes associated with it to uniquely identify the user
|
||||
|
||||
and so on.
|
||||
|
||||
For this assessment, we're looking at six basic control families:
|
||||
|
||||
- Networking
|
||||
- Cryptography
|
||||
- Secrets Management
|
||||
- Authentication
|
||||
- Authorization (Access Control)
|
||||
- Multi-tenancy Isolation
|
||||
|
||||
Obviously we can skip control families as "not applicable" in the event that the component does not require it. For example,
|
||||
something with the sole purpose of interacting with the local file system may have no meaningful Networking component; this
|
||||
isn't a weakness, it's simply "not applicable."
|
||||
|
||||
For each control family we want to ask:
|
||||
|
||||
- What does the component do for this control?
|
||||
- What sorts of data passes through that control?
|
||||
- for example, a component may have sensitive data (Secrets Management), but that data never leaves the component's storage via Networking
|
||||
- What can attacker do with access to this component?
|
||||
- What's the simplest attack against it?
|
||||
- Are there mitigations that we recommend (i.e. "Always use an interstitial firewall")?
|
||||
- What happens if the component stops working (via DoS or other means)?
|
||||
- Have there been similar vulnerabilities in the past? What were the mitigations?
|
||||
|
||||
# Threat Scenarios
|
||||
|
||||
- An External Attacker without access to the client application
|
||||
- An External Attacker with valid access to the client application
|
||||
- An Internal Attacker with access to cluster
|
||||
- A Malicious Internal User
|
||||
|
||||
## Networking
|
||||
|
||||
## Cryptography
|
||||
|
||||
## Secrets Management
|
||||
|
||||
## Authentication
|
||||
|
||||
- by default Kubernetes doesn't use two-way TLS to the etcd cluster, which would be the most secure (combined with IP restrictions so that stolen creds can't be reused on new infrastructure)
|
||||
|
||||
## Authorization
|
||||
|
||||
## Multi-tenancy Isolation
|
||||
|
||||
## Summary
|
||||
|
||||
# Recommendations
|
||||
|
|
@ -0,0 +1,18 @@
|
|||
# Meeting notes
|
||||
|
||||
- CCM per cloud provider
|
||||
- same host as kube-apiserver
|
||||
- caches live in memory
|
||||
- refresh cache, but can be forced to by request
|
||||
- Controller manager attempts to use PoLA, but the service account controller has permission to write to it's own policies
|
||||
- Cloud controller (routes, IPAM, &c.) can talk to external resources
|
||||
- CCM/KCM have no notion of multi-tenant, and there are implications going forward
|
||||
- Deployments across namespace
|
||||
- cloud controller has access to cloud credentials (passed in by various means, as we saw in the code)
|
||||
- CCM is a reference implementation, meant to separate out other company's code
|
||||
- So Amazon doesn't need to have Red Hat's code running, &c.
|
||||
- shared acache across all controllers
|
||||
- [FINDING] separate out high privileged controllers from lower privileged ones, so there's no confused deputy
|
||||
- single binary for controller
|
||||
- if you can trick the service account controller into granting access to things you shouldn't (for example) that would be problematic
|
||||
- make a "privileged controller manager" which bundles high and low-privileged controllers, and adds another trust boundary
|
||||
|
|
@ -0,0 +1,187 @@
|
|||
# Overview
|
||||
|
||||
- Component: kube-apiserver
|
||||
- Owner(s): [sig-api-machinery](https://github.com/kubernetes/community/tree/master/sig-api-machinery)
|
||||
- SIG/WG(s) at meeting:
|
||||
- Service Data Classification: Critical (technically, it isn't needed, but most clusters will use it extensively)
|
||||
- Highest Risk Impact:
|
||||
|
||||
# Service Notes
|
||||
|
||||
The portion should walk through the component and discuss connections, their relevant controls, and generally lay out how the component serves its relevant function. For example
|
||||
a component that accepts an HTTP connection may have relevant questions about channel security (TLS and Cryptography), authentication, authorization, non-repudiation/auditing,
|
||||
and logging. The questions aren't the *only* drivers as to what may be spoken about, the questions are meant to drive what we discuss and keep things on task for the duration
|
||||
of a meeting/call.
|
||||
|
||||
## How does the service work?
|
||||
|
||||
- RESTful API server
|
||||
- made up of multiple subcomponents:
|
||||
- authenticators
|
||||
- authorizers
|
||||
- admission controllers
|
||||
- resource validators
|
||||
- users issue a request, which is authenticated via one (or more) plugins
|
||||
- the requests is then authorized by one or more authorizers
|
||||
- it is then potentially modified and validated by an admission controller
|
||||
- resource validation that validates the object, stores it in etcd, and responds
|
||||
- clients issue HTTP requests (via TLS ala HTTPS) to "watch" resources and poll for changes from the server; for example:
|
||||
1. a client updates a pod definition via `kubectl` and a `POST` request
|
||||
1. the scheduler is "watching" for pod updates via an HTTP watch request to retrieve new pods
|
||||
1. the scheduler then update the pod list via a `POST` to the kube-apiserver
|
||||
1. a node's `kubelet` retrieves a list of pods assigned to it via an HTTP watch request
|
||||
1. the node's `kubelet` then update the running pod list on the kube-apiserver
|
||||
|
||||
## Are there any subcomponents or shared boundaries?
|
||||
|
||||
Yes
|
||||
|
||||
- Controllers technically run on the kube-apiserver
|
||||
- the various subcomponents (authenticators, authorizers, and so on) run on the kube-apiserver
|
||||
|
||||
additionally, depending on the configuration there may be any number of other Master Control Pane components running on the same phyical/logical host
|
||||
|
||||
## What communications protocols does it use?
|
||||
|
||||
- Communcations to the kube-apiserver use HTTPS and various authentication mechanisms
|
||||
- Communications from the kube-apiserver to etcd use HTTPS, with optional client-side (two-way) TLS
|
||||
- Communications from the kube-apiserver to kubelets can use HTTP or HTTPS, the latter is without validation by default (find this again in the docs)
|
||||
|
||||
## Where does it store data?
|
||||
|
||||
- Most data is stored in etcd, mainly under `/registry`
|
||||
- Some data is obviously stored on the local host, to bootstrap the connection to etcd
|
||||
|
||||
## What is the most sensitive data it stores?
|
||||
|
||||
- Not much sensitive is directly stored on kube-apiserver
|
||||
- However, all sensitive data within the system (save for in MCP-less setups) is processed and transacted via the kube-apiserver
|
||||
|
||||
## How is that data stored?
|
||||
|
||||
- On etcd, with the level of protection requested by the user
|
||||
- looks like encryption [is a command line flag](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/#configuration-and-determining-whether-encryption-at-rest-is-already-enabled)
|
||||
|
||||
# Meeting notes
|
||||
|
||||
- web hooks: kube-apiserver can call eternal resources
|
||||
- authorization webhook (for when you wish to auth a request without setting up a new authorizer)
|
||||
- images, other resources
|
||||
- [FINDING] supports HTTP
|
||||
- Aggregate API server // Aggregator
|
||||
- for adding externisbility resources
|
||||
- a type of CRD, basically
|
||||
- component status -> reaches out to every component on the cluster
|
||||
- Network proxy: restrict outbound connections from kube-apiserver (currently no restriction)
|
||||
- honestly a weakness: no egress filtering
|
||||
- Business logic in controllers, but kube-apiserver is info
|
||||
- cloud prociders, auth, &c
|
||||
- sharding by group version kind, put all KVKs into the same etcd
|
||||
- listeners: insecure and secure
|
||||
- check if insecure is configured by default
|
||||
- would be a finding if so
|
||||
- Not comfortable doing true multi-tenant on k8s
|
||||
- multi-single tenants (as in, if Pepsi wants to have marketing & accounting that's fine, but not Coke & Pepsi on the same cluster)
|
||||
- Best way to restrict access to kube-apiserver
|
||||
- and working on a proxy as noted above
|
||||
- kube-apiserver is the root CA for *at least two* PKIs:
|
||||
- two CAs, but not on by default w/o flags (check what happens w/o two CAs...)
|
||||
- that would be a finding, if you can cross CAs really
|
||||
- TLS (multiple domains):
|
||||
- etcd -> kube-apiserver
|
||||
- the other is webhooks/kublet/components...
|
||||
- check secrets: can you tell k8s to encrypt a secret but not provide the flag? what does it do?
|
||||
- Alt route for secrets: volumes, write to a volume, then mount
|
||||
- Can't really do much about that, since it's opaque to the kube-apiserver
|
||||
- ConfigMap: people can stuff secrets into ConfigMaps
|
||||
- untyped data blob
|
||||
- cannot encrypt
|
||||
- recommend moving away from ConfigMaps
|
||||
- Logging to var log
|
||||
- resource names in logs (namespace, secret name, &c). Can be sensitive
|
||||
- [FINDING] no logs by default who did what
|
||||
- need to turn on auditing for that
|
||||
- look at metrics as well, similar to CRDs
|
||||
- Data Validation
|
||||
- can have admission controller, webhooks, &c.
|
||||
- everything goes through validation
|
||||
- Session
|
||||
- upgrade to HTTP/2, channel, or SPDY
|
||||
- JWT is long lived (we know)
|
||||
- Certain requests like proxy and logs require upgrade to channels
|
||||
- look at k8s enhancement ... kube-apiserver dot md
|
||||
|
||||
# Data Dictionary
|
||||
|
||||
| Name | Classification/Sensitivity | Comments |
|
||||
| :--: | :--: | :--: |
|
||||
| Data | Goes | Here |
|
||||
|
||||
# Control Families
|
||||
|
||||
These are the areas of controls that we're interested in based on what the audit working group selected.
|
||||
|
||||
When we say "controls," we mean a logical section of an application or system that handles a security requirement. Per CNSSI:
|
||||
|
||||
> The management, operational, and technical controls (i.e., safeguards or countermeasures) prescribed for an information system to protect the confidentiality, integrity, and availability of the system and its information.
|
||||
|
||||
For example, an system may have authorization requirements that say:
|
||||
|
||||
- users must be registered with a central authority
|
||||
- all requests must be verified to be owned by the requesting user
|
||||
- each account must have attributes associated with it to uniquely identify the user
|
||||
|
||||
and so on.
|
||||
|
||||
For this assessment, we're looking at six basic control families:
|
||||
|
||||
- Networking
|
||||
- Cryptography
|
||||
- Secrets Management
|
||||
- Authentication
|
||||
- Authorization (Access Control)
|
||||
- Multi-tenancy Isolation
|
||||
|
||||
Obviously we can skip control families as "not applicable" in the event that the component does not require it. For example,
|
||||
something with the sole purpose of interacting with the local file system may have no meaningful Networking component; this
|
||||
isn't a weakness, it's simply "not applicable."
|
||||
|
||||
For each control family we want to ask:
|
||||
|
||||
- What does the component do for this control?
|
||||
- What sorts of data passes through that control?
|
||||
- for example, a component may have sensitive data (Secrets Management), but that data never leaves the component's storage via Networking
|
||||
- What can attacker do with access to this component?
|
||||
- What's the simplest attack against it?
|
||||
- Are there mitigations that we recommend (i.e. "Always use an interstitial firewall")?
|
||||
- What happens if the component stops working (via DoS or other means)?
|
||||
- Have there been similar vulnerabilities in the past? What were the mitigations?
|
||||
|
||||
# Threat Scenarios
|
||||
|
||||
- An External Attacker without access to the client application
|
||||
- An External Attacker with valid access to the client application
|
||||
- An Internal Attacker with access to cluster
|
||||
- A Malicious Internal User
|
||||
|
||||
## Networking
|
||||
|
||||
- in the version of k8s we are testing, no outbound limits on external connections
|
||||
|
||||
## Cryptography
|
||||
|
||||
- Not encrypting secrets in etcd by default
|
||||
- requiring [a command line flag](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/#configuration-and-determining-whether-encryption-at-rest-is-already-enabled)
|
||||
- SUpports HTTP for Webhooks and comopnent status
|
||||
|
||||
## Secrets Management
|
||||
|
||||
## Authentication
|
||||
|
||||
## Authorization
|
||||
|
||||
## Multi-tenancy Isolation
|
||||
|
||||
## Summary
|
||||
|
||||
# Recommendations
|
||||
|
|
@ -0,0 +1,227 @@
|
|||
# Overview
|
||||
|
||||
- Component: kube-proxy
|
||||
- Owner(s): [sig-network](https://github.com/kubernetes/community/tree/master/sig-network)
|
||||
- SIG/WG(s) at meeting:
|
||||
- Service Data Classification: Medium
|
||||
- Highest Risk Impact:
|
||||
|
||||
# Service Notes
|
||||
|
||||
The portion should walk through the component and discuss connections, their relevant controls, and generally lay out how the component serves its relevant function. For example
|
||||
a component that accepts an HTTP connection may have relevant questions about channel security (TLS and Cryptography), authentication, authorization, non-repudiation/auditing,
|
||||
and logging. The questions aren't the *only* drivers as to what may be spoken about, the questions are meant to drive what we discuss and keep things on task for the duration
|
||||
of a meeting/call.
|
||||
|
||||
## How does the service work?
|
||||
|
||||
- kubeproxy has several main modes of operation:
|
||||
- as a literal network proxy, handling networking between nodes
|
||||
- as a bridge between Container Network Interface (CNI) which handles the actual networking and the host operating system
|
||||
- `iptables` mode
|
||||
- `ipvs` mode
|
||||
- two Microsoft Windows-specific modes (not covered by the RRA)
|
||||
- in any of these modes, kubeproxy interfaces with the host's routing table so as to achieve a seamless, flat network across the kubernetes cluster
|
||||
|
||||
## Are there any subcomponents or shared boundaries?
|
||||
|
||||
Yes.
|
||||
|
||||
- Similar to kubelet, kube-proxy run's on the node, with an implicit trust boundary between Worker components and Container components (i.e. pods)
|
||||
|
||||
## What communications protocols does it use?
|
||||
|
||||
- Direct IPC to `iptables` or `ipvs`
|
||||
- HTTPS to the kube-apiserver
|
||||
- HTTP Healthz port (which is a literal counter plus a `200 Ok` response)
|
||||
|
||||
## Where does it store data?
|
||||
|
||||
Minimal data should be stored by kube-proxy itself, this should mainly be handled by kubelet and some file system configuration
|
||||
|
||||
## What is the most sensitive data it stores?
|
||||
|
||||
N/A
|
||||
|
||||
## How is that data stored?
|
||||
|
||||
N/A
|
||||
|
||||
# Data Dictionary
|
||||
|
||||
| Name | Classification/Sensitivity | Comments |
|
||||
| :--: | :--: | :--: |
|
||||
| Data | Goes | Here |
|
||||
|
||||
# Control Families
|
||||
|
||||
These are the areas of controls that we're interested in based on what the audit working group selected.
|
||||
|
||||
When we say "controls," we mean a logical section of an application or system that handles a security requirement. Per CNSSI:
|
||||
|
||||
> The management, operational, and technical controls (i.e., safeguards or countermeasures) prescribed for an information system to protect the confidentiality, integrity, and availability of the system and its information.
|
||||
|
||||
For example, an system may have authorization requirements that say:
|
||||
|
||||
- users must be registered with a central authority
|
||||
- all requests must be verified to be owned by the requesting user
|
||||
- each account must have attributes associated with it to uniquely identify the user
|
||||
|
||||
and so on.
|
||||
|
||||
For this assessment, we're looking at six basic control families:
|
||||
|
||||
- Networking
|
||||
- Cryptography
|
||||
- Secrets Management
|
||||
- Authentication
|
||||
- Authorization (Access Control)
|
||||
- Multi-tenancy Isolation
|
||||
|
||||
Obviously we can skip control families as "not applicable" in the event that the component does not require it. For example,
|
||||
something with the sole purpose of interacting with the local file system may have no meaningful Networking component; this
|
||||
isn't a weakness, it's simply "not applicable."
|
||||
|
||||
For each control family we want to ask:
|
||||
|
||||
- What does the component do for this control?
|
||||
- What sorts of data passes through that control?
|
||||
- for example, a component may have sensitive data (Secrets Management), but that data never leaves the component's storage via Networking
|
||||
- What can attacker do with access to this component?
|
||||
- What's the simplest attack against it?
|
||||
- Are there mitigations that we recommend (i.e. "Always use an interstitial firewall")?
|
||||
- What happens if the component stops working (via DoS or other means)?
|
||||
- Have there been similar vulnerabilities in the past? What were the mitigations?
|
||||
|
||||
# Threat Scenarios
|
||||
|
||||
- An External Attacker without access to the client application
|
||||
- An External Attacker with valid access to the client application
|
||||
- An Internal Attacker with access to cluster
|
||||
- A Malicious Internal User
|
||||
|
||||
## Networking
|
||||
|
||||
- kube-proxy is actually five programs
|
||||
- proxy: mostly deprecated, but a literal proxy, in that it intercepts requests and proxies them to backend services
|
||||
- IPVS/iptables: very similar modes, handle connecting virtual IPs (VIPs) and the like via low-level routing (the preferred mode)
|
||||
- two Windows-specific modes (out of scope for this discussion, but if there are details we can certainly add them)
|
||||
|
||||
Node ports:
|
||||
|
||||
- captures traffic from Host IP
|
||||
- shuffles to backend (used for building load balancers)
|
||||
|
||||
- kube-proxy shells out to `iptables` or `ipvs`
|
||||
- Also uses a netlink socket for IPVS (netlink are similar to Unix Domain Sockets)
|
||||
- *Also* shells out to `ipset` under certain circumstances for IPVS (building sets of IPs and such)
|
||||
|
||||
|
||||
### User space proxy
|
||||
|
||||
Setup:
|
||||
|
||||
1. Connect to the kube-apiserver
|
||||
1. Watch the API server for services/endpoints/&c
|
||||
1. Build in-memory caching map: for services, for every port a service maps, open a port, write iptables rule for VIP & Virt Port
|
||||
1. Watch for updates of services/endpoints/&c
|
||||
|
||||
when a consumer connects to the port:
|
||||
|
||||
1. Service is running VIP:VPort
|
||||
1. Root NS -> iptable -> kube-proxy port
|
||||
1. look at the src/dst port, check the map, pick a service on that port at random (if that fails, try another until either success or a retry count has exceeded)
|
||||
1. Shuffle bytes back and forth between backend service and client until termination or failure
|
||||
|
||||
### iptables
|
||||
|
||||
1. Same initial setup (sans opening a port directly)
|
||||
1. iptables restore command set
|
||||
1. giant string of services
|
||||
1. User VIP -> Random Backend -> Rewrite packets (at the kernel level, so kube-proxy never sees the data)
|
||||
1. At the end of the sync loop, write (write in batches to avoid iptables contentions)
|
||||
1. no more routing table touches until service updates (from watching kube-apiserver or a time out, expanded below)
|
||||
|
||||
**NOTE**: rate limited (bounded frequency) updates:
|
||||
- no later than 10 minutes by default
|
||||
- no sooner than 15s by default (if there are no service map updates)
|
||||
|
||||
this point came out of the following question: is having access to kube-proxy *worse* than having root access to the host machine?
|
||||
|
||||
### ipvs
|
||||
|
||||
1. Same setup as iptables & proxy mode
|
||||
1. `ipvsadm` and `ipset` commands instead of `iptables`
|
||||
1. This does have some strange changes:
|
||||
- ip address needs a dummy adapter
|
||||
- !NOTE Any service bound to 0.0.0.0 are also bound to _all_ adapters
|
||||
- somewhat expected because 0.0.0.0, but can still lead to interesting behavior
|
||||
|
||||
### concern points within networking
|
||||
|
||||
- !NOTE: ARP table attacks (such as if someone has `CAP_NET_RAW` in a container or host access) can impact kube-proxy
|
||||
- Endpoint selection is namespace & pod-based, so injection could overwrite (I don't think this is worth a finding/note because kube-apiserver is the arbiter of truth)
|
||||
- !FINDING (but low...): POD IP Reuse: (factor of 2 x max) cause a machine to churn thru IPS, you could cause a kube-proxy to forward ports to your pod if you win the race condition.
|
||||
- this would be limited to the window of routing updates
|
||||
- however, established connections would remain
|
||||
- kube-apiserver could be the arbiter of routing, but that may require more watch and connection to the central component
|
||||
- [editor] I think just noting this potential issue and maybe warning on it in kube-proxy logs would be enough
|
||||
|
||||
### with root access?
|
||||
|
||||
Access to kube-proxy is mostly the same as root access
|
||||
|
||||
- set syscalls, route local, &c could gobble memory
|
||||
- Node/VIP level
|
||||
- Recommend `CAP_NET_BIND` (bind to low ports, don't need root for certain users) for containers/pods, alleviate concerns there
|
||||
- Can map low ports to high ports in kube-proxy as well, but mucks with anything that pretends to be a VIP
|
||||
- LB forwards packets to service without new connection (based on srcport)
|
||||
- 2-hop LB, can't do direct LB
|
||||
|
||||
## Cryptography
|
||||
|
||||
- kube-proxy itself does not handle cryptography other than the TLS connection to kube-apiserver
|
||||
|
||||
## Secrets Management
|
||||
|
||||
- kube-proxy itself does not handle secrets, but rather only consumes credentials from the command line (like all other k8s components)
|
||||
|
||||
## Authentication
|
||||
|
||||
- kube-proxy does not handle any authentication other than credentials to the kube-apiserver
|
||||
|
||||
## Authorization
|
||||
|
||||
- kube-proxy does not handle any authorization; the arbiters of authorization are kubelet and kube-proxy
|
||||
|
||||
## Multi-tenancy Isolation
|
||||
|
||||
- kube-proxy does not currently segment clients from one another, as clients on the same pod/host must use the same iptables/ipvs configuration
|
||||
- kube-proxy does have conception of namespaces, but currently avoids enforcing much at that level
|
||||
- routes still must be added to iptables or the like
|
||||
- iptables contention could be problematic
|
||||
- much better to handle at higher-level components, namely kube-apiserver and kube-proxy
|
||||
|
||||
## Logging
|
||||
|
||||
- stderr directed to a file
|
||||
- same as with kubelet
|
||||
- !FINDING (but same as all other components) logs namespaces, service names (same as every other service)
|
||||
|
||||
# Additional Notes
|
||||
|
||||
## kubelet to iptables
|
||||
|
||||
- per pod network management
|
||||
- pods can request a host port, docker style
|
||||
- kubenet and CNI plugins
|
||||
- kubenet uses CNI
|
||||
- setup kubenet iptable to map ports to a single pod
|
||||
- overly broad, should be appended to iptables list
|
||||
- all local IPs to the host
|
||||
|
||||
!FINDING: don't use host ports, they can cause problems with services and such; we may recommend deprecating them
|
||||
|
||||
## Summary
|
||||
|
||||
# Recommendations
|
||||
|
|
@ -0,0 +1,162 @@
|
|||
# Overview
|
||||
|
||||
- Component: kube-scheduler
|
||||
- Owner(s): [sig-scheduling](https://github.com/kubernetes/community/tree/master/sig-scheduling)
|
||||
- SIG/WG(s) at meeting:
|
||||
- Service Data Classifjcation: Moderate (the scheduler adds pods to nodes, but will not remove pods, for the most part)
|
||||
- Highest Risk Impact:
|
||||
|
||||
# Service Notes
|
||||
|
||||
The portion should walk through the component and discuss connections, their relevant controls, and generally lay out how the component serves its relevant function. For example
|
||||
a component that accepts an HTTP connection may have relevant questions about channel security (TLS and Cryptography), authentication, authorization, non-repudiation/auditing,
|
||||
and logging. The questions aren't the *only* drivers as to what may be spoken about, the questions are meant to drive what we discuss and keep things on task for the duration
|
||||
of a meeting/call.
|
||||
|
||||
## How does the service work?
|
||||
|
||||
- Similar to most other components:
|
||||
1. Watches for unscheduled/new pods
|
||||
1. Watches nodes with and their resource constraints
|
||||
1. Chooses a node, via various mechanisms, to allocate based on best fit of resource requirements
|
||||
1. Updates the pod spec on the kube-apiserver
|
||||
1. that update is then retrieved by the node, which is also Watching components via the kube-apiserver
|
||||
- there may be multiple schedulers with various names, and parameters (such as pod-specific schedulers)
|
||||
|
||||
- !NOTE schedulers are coöperative
|
||||
- !NOTE schedulers are *supposed* to honor the name, but need not
|
||||
- Interesting note, makes the huge list of schedulers DoS interesting
|
||||
- !NOTE idea there was to add a *huge* number of pods to be scheduled that are associated with an poorly named scheduler
|
||||
- !NOTE peopoe shouldn't request specific schedulers in podspec, rather, there should be some webhook to process that
|
||||
- !NOTE team wasn't sure what would happen with large number of pods to be scheduled
|
||||
|
||||
## Are there any subcomponents or shared boundaries?
|
||||
|
||||
Yes
|
||||
|
||||
- there may be multiple schedulers on the same MCP host
|
||||
- schedulers may run on the same host as the API server
|
||||
|
||||
## What communications protocols does it use?
|
||||
|
||||
- standard HTTPS + auth (chosen by the cluster)
|
||||
|
||||
## Where does it store data?
|
||||
|
||||
- most should be stored in etcd (via kube-apiserver)
|
||||
- some data will be stored on command line (configuration options) or on the file system (certificate paths for authentication)
|
||||
|
||||
## What is the most sensitive data it stores?
|
||||
|
||||
- No direct storage
|
||||
|
||||
## How is that data stored?
|
||||
|
||||
- N/A
|
||||
|
||||
# Data Dictionary
|
||||
|
||||
| Name | Classification/Sensitivity | Comments |
|
||||
| :--: | :--: | :--: |
|
||||
| Data | Goes | Here |
|
||||
|
||||
# Control Families
|
||||
|
||||
These are the areas of controls that we're interested in based on what the audit working group selected.
|
||||
|
||||
When we say "controls," we mean a logical section of an application or system that handles a security requirement. Per CNSSI:
|
||||
|
||||
> The management, operational, and technical controls (i.e., safeguards or countermeasures) prescribed for an information system to protect the confidentiality, integrity, and availability of the system and its information.
|
||||
|
||||
For example, an system may have authorization requirements that say:
|
||||
|
||||
- users must be registered with a central authority
|
||||
- all requests must be verified to be owned by the requesting user
|
||||
- each account must have attributes associated with it to uniquely identify the user
|
||||
|
||||
and so on.
|
||||
|
||||
For this assessment, we're looking at six basic control families:
|
||||
|
||||
- Networking
|
||||
- Cryptography
|
||||
- Secrets Management
|
||||
- Authentication
|
||||
- Authorization (Access Control)
|
||||
- Multi-tenancy Isolation
|
||||
|
||||
Obviously we can skip control families as "not applicable" in the event that the component does not require it. For example,
|
||||
something with the sole purpose of interacting with the local file system may have no meaningful Networking component; this
|
||||
isn't a weakness, it's simply "not applicable."
|
||||
|
||||
For each control family we want to ask:
|
||||
|
||||
- What does the component do for this control?
|
||||
- What sorts of data passes through that control?
|
||||
- for example, a component may have sensitive data (Secrets Management), but that data never leaves the component's storage via Networking
|
||||
- What can attacker do with access to this component?
|
||||
- What's the simplest attack against it?
|
||||
- Are there mitigations that we recommend (i.e. "Always use an interstitial firewall")?
|
||||
- What happens if the component stops working (via DoS or other means)?
|
||||
- Have there been similar vulnerabilities in the past? What were the mitigations?
|
||||
|
||||
# Threat Scenarios
|
||||
|
||||
- An External Attacker without access to the client application
|
||||
- An External Attacker with valid access to the client application
|
||||
- An Internal Attacker with access to cluster
|
||||
- A Malicious Internal User
|
||||
|
||||
## Networking
|
||||
|
||||
- only talks to kube-apiserver
|
||||
- colocated on the same host generally as kube-apiserver, but needn't be
|
||||
- has a web server (HTTP)
|
||||
- !FINDING: same HTTP server finding as all other components
|
||||
- metrics endpoint: qps, scheduling latency, &c
|
||||
- healthz endpoint, which is just a 200 Ok response
|
||||
- by default doesn't verify cert (maybe)
|
||||
|
||||
## Cryptography
|
||||
|
||||
- None
|
||||
|
||||
## Secrets Management
|
||||
|
||||
- Logs is the only persistence mechanism
|
||||
- !FINDING (to be added to all the other "you expose secrets in env and CLI" finding locations) auth token/cred passed in via CLI
|
||||
|
||||
## Authentication
|
||||
|
||||
- no authN really
|
||||
- pods, nodes, related objects; doesn't deal in authN
|
||||
- unaware of any service/user accounts
|
||||
|
||||
## Authorization
|
||||
|
||||
- schedluinc concepts protected by authZ
|
||||
- quotas
|
||||
- priority classes
|
||||
- &c
|
||||
- this authZ is not enforced by scheduler, however, enforced by kube-apiserver
|
||||
|
||||
## Multi-tenancy Isolation
|
||||
|
||||
- tenant: different users of workloads that don't want to trust one another
|
||||
- namespaces are usually the boundaries
|
||||
- affinity/anti-affinity for namespace
|
||||
- scheduler doesn't have data plan access
|
||||
- can have noisy neighbory problem
|
||||
- is that the scheduler's issue?
|
||||
- not sure
|
||||
- namspace agnostic
|
||||
- can use priority classes which can be RBAC'd to a specific namespace, like kube-system
|
||||
- does not handle tenant fairness, handles priorty class fairness
|
||||
- no visibility into network boundary or usage information
|
||||
- no cgroup for network counts
|
||||
- !FINDING anti-affinity can be abused: only I can have this one host, no one else, applicable from `kubectl`
|
||||
- !NOTE no backoff process for scheduler to reschedule a rejected pod by the kublet; the replicaset controller can create a tightloop (RSC -> Scheduler -> Kubelet -> Reject -> RSC...)
|
||||
|
||||
## Summary
|
||||
|
||||
# Recommendations
|
||||
|
|
@ -0,0 +1,180 @@
|
|||
# Overview
|
||||
|
||||
- Component: kubelet
|
||||
- Owner(s): [sig-node](https://github.com/kubernetes/community/tree/master/sig-node)
|
||||
- SIG/WG(s) at meeting:
|
||||
- Service Data Classification: High
|
||||
- Highest Risk Impact:
|
||||
|
||||
# Service Notes
|
||||
|
||||
The portion should walk through the component and discuss connections, their relevant controls, and generally lay out how the component serves its relevant function. For example
|
||||
a component that accepts an HTTP connection may have relevant questions about channel security (TLS and Cryptography), authentication, authorization, non-repudiation/auditing,
|
||||
and logging. The questions aren't the *only* drivers as to what may be spoken about, the questions are meant to drive what we discuss and keep things on task for the duration
|
||||
of a meeting/call.
|
||||
|
||||
## How does the service work?
|
||||
|
||||
- `kubelet` isses a watch request on the `kube-apiserver`
|
||||
- `kubelet` watches for pod allocations assigned to the node the kubelet is currently running on
|
||||
- when a new pod has been allocated for the kubelet's host, it retrieve the pod spec, and interacts with the Container Runtime via local Interprocess Communication to run the container
|
||||
- Kubelet also handles:
|
||||
- answering log requests from the kube-apiserver
|
||||
- monitoring pod health for failures
|
||||
- working with the Container Runtime to deschedule pods when the pod has been deleted
|
||||
- updating the kube-apiserver with host status (for use by the scheduler)
|
||||
|
||||
## Are there any subcomponents or shared boundaries?
|
||||
|
||||
Yes.
|
||||
|
||||
- Technically, kubelet runs on the same host as the Container Runtime and kubeproxy
|
||||
- There is a Trust Zone boundary between the Container Runtime and the kubelet
|
||||
|
||||
## What communications protocols does it use?
|
||||
|
||||
- HTTPS with certificate validation and some authentication mechanism for communication with the kube-apiserver as a client
|
||||
- HTTPS without certificate validation by default
|
||||
|
||||
## Where does it store data?
|
||||
|
||||
- kubelet itself should not store much data
|
||||
- kubelet can be run in an "apiserver-less mode" that loads pod manifests from the file system
|
||||
- most data should be retrieved from the kube-apiserver via etcd
|
||||
- authentication credentials for the kube-apiserver may be stored on the file system or in memory (both in CLI parameter as well as actual program memory) for the duration of execution
|
||||
|
||||
## What is the most sensitive data it stores?
|
||||
|
||||
- authentication credentials are stored in memory or are out of scope
|
||||
|
||||
## How is that data stored?
|
||||
|
||||
N/A
|
||||
|
||||
# Data Dictionary
|
||||
|
||||
| Name | Classification/Sensitivity | Comments |
|
||||
| :--: | :--: | :--: |
|
||||
| Data | Goes | Here |
|
||||
|
||||
# Control Families
|
||||
|
||||
These are the areas of controls that we're interested in based on what the audit working group selected.
|
||||
|
||||
When we say "controls," we mean a logical section of an application or system that handles a security requirement. Per CNSSI:
|
||||
|
||||
> The management, operational, and technical controls (i.e., safeguards or countermeasures) prescribed for an information system to protect the confidentiality, integrity, and availability of the system and its information.
|
||||
|
||||
For example, an system may have authorization requirements that say:
|
||||
|
||||
- users must be registered with a central authority
|
||||
- all requests must be verified to be owned by the requesting user
|
||||
- each account must have attributes associated with it to uniquely identify the user
|
||||
|
||||
and so on.
|
||||
|
||||
For this assessment, we're looking at six basic control families:
|
||||
|
||||
- Networking
|
||||
- Cryptography
|
||||
- Secrets Management
|
||||
- Authentication
|
||||
- Authorization (Access Control)
|
||||
- Multi-tenancy Isolation
|
||||
|
||||
Obviously we can skip control families as "not applicable" in the event that the component does not require it. For example,
|
||||
something with the sole purpose of interacting with the local file system may have no meaningful Networking component; this
|
||||
isn't a weakness, it's simply "not applicable."
|
||||
|
||||
For each control family we want to ask:
|
||||
|
||||
- What does the component do for this control?
|
||||
- What sorts of data passes through that control?
|
||||
- for example, a component may have sensitive data (Secrets Management), but that data never leaves the component's storage via Networking
|
||||
- What can attacker do with access to this component?
|
||||
- What's the simplest attack against it?
|
||||
- Are there mitigations that we recommend (i.e. "Always use an interstitial firewall")?
|
||||
- What happens if the component stops working (via DoS or other means)?
|
||||
- Have there been similar vulnerabilities in the past? What were the mitigations?
|
||||
|
||||
# Threat Scenarios
|
||||
|
||||
- An External Attacker without access to the client application
|
||||
- An External Attacker with valid access to the client application
|
||||
- An Internal Attacker with access to cluster
|
||||
- A Malicious Internal User
|
||||
|
||||
## Networking
|
||||
|
||||
- Post 10250: read/write, authenticated
|
||||
- Port 10255: read-only, unauthenticated
|
||||
- cadvisor uses this, going to be deprecated
|
||||
- 10248: healthz, unauth'd
|
||||
- static pod manifest directory
|
||||
- Static pod fetch via HTTP(S)
|
||||
|
||||
### Routes:
|
||||
|
||||
- Auth filter on API, for 10250
|
||||
- delegated to apiserver, subject access review, HTTPS request
|
||||
- `/pods` podspec on node -> leaks data
|
||||
- `/healthz`
|
||||
- `/spec`
|
||||
- `/stats-{cpu, mem, &c}`
|
||||
- on 10250 only:
|
||||
- `/exec`
|
||||
- `/attach`
|
||||
- `portforward`
|
||||
- `/kube-auth`
|
||||
- `/debug-flags`
|
||||
- `/cri/{exec, attach, portforward}`
|
||||
|
||||
### Findings:
|
||||
|
||||
- !FINDING: 10255 is unauthenticated and leaks secrets
|
||||
- !FINDING: 10255/10248
|
||||
- !FINDING: 10250 is self-signed TLS
|
||||
|
||||
## Cryptography
|
||||
|
||||
- None
|
||||
|
||||
## Secrets Management
|
||||
|
||||
- returned from kube-apiserver unencrypted
|
||||
- in memory cache
|
||||
- if pod mounts disk, written to tmpfs
|
||||
- !FINDING (already captured) ENV vars can expose secrets
|
||||
- configmaps are treated like secrets by kubelet
|
||||
- !FINDING keynames and secret names may be logged
|
||||
- maintains its own certs, secrets, bootstrap credential
|
||||
- bootstrap: initial cert used to issue CSR to kube-apiserver
|
||||
- !NOTE certs are written to disk unencrypted
|
||||
- !FINDING bootstrap cert may be long lived, w/o a TTL
|
||||
|
||||
## Authentication
|
||||
|
||||
- delegated to kube-apiserver, via HTTPS request, with subject access review
|
||||
- two-way TLS by default (we believe)
|
||||
- token auth
|
||||
- bearer token
|
||||
- passed to request to API server
|
||||
- "token review"
|
||||
- kube-apiserver responds w/ ident
|
||||
- response is boolean (yes/no is this a user) and username/uid/groups/arbitrary data as a tuple
|
||||
- no auditing on kublet, but logged on kube-apiserver
|
||||
|
||||
## Authorization
|
||||
|
||||
- delegated to kube-apiserver
|
||||
|
||||
## Multi-tenancy Isolation
|
||||
|
||||
- kube-apiserver is the arbiter
|
||||
- kubelet doesn't know namespaces really
|
||||
- every pod is a separate tenant
|
||||
- pods are security boundaries
|
||||
|
||||
## Summary
|
||||
|
||||
# Recommendations
|
||||
|
|
@ -0,0 +1,95 @@
|
|||
# Overview
|
||||
|
||||
- Component:
|
||||
- Owner(s):
|
||||
- SIG/WG(s) at meeting:
|
||||
- Service Data Classification:
|
||||
- Highest Risk Impact:
|
||||
|
||||
# Service Notes
|
||||
|
||||
The portion should walk through the component and discuss connections, their relevant controls, and generally lay out how the component serves its relevant function. For example
|
||||
a component that accepts an HTTP connection may have relevant questions about channel security (TLS and Cryptography), authentication, authorization, non-repudiation/auditing,
|
||||
and logging. The questions aren't the *only* drivers as to what may be spoken about, the questions are meant to drive what we discuss and keep things on task for the duration
|
||||
of a meeting/call.
|
||||
|
||||
## How does the service work?
|
||||
|
||||
## Are there any subcomponents or shared boundaries?
|
||||
|
||||
## What communications protocols does it use?
|
||||
|
||||
## Where does it store data?
|
||||
|
||||
## What is the most sensitive data it stores?
|
||||
|
||||
## How is that data stored?
|
||||
|
||||
# Data Dictionary
|
||||
|
||||
| Name | Classification/Sensitivity | Comments |
|
||||
| :--: | :--: | :--: |
|
||||
| Data | Goes | Here |
|
||||
|
||||
# Control Families
|
||||
|
||||
These are the areas of controls that we're interested in based on what the audit working group selected.
|
||||
|
||||
When we say "controls," we mean a logical section of an application or system that handles a security requirement. Per CNSSI:
|
||||
|
||||
> The management, operational, and technical controls (i.e., safeguards or countermeasures) prescribed for an information system to protect the confidentiality, integrity, and availability of the system and its information.
|
||||
|
||||
For example, an system may have authorization requirements that say:
|
||||
|
||||
- users must be registered with a central authority
|
||||
- all requests must be verified to be owned by the requesting user
|
||||
- each account must have attributes associated with it to uniquely identify the user
|
||||
|
||||
and so on.
|
||||
|
||||
For this assessment, we're looking at six basic control families:
|
||||
|
||||
- Networking
|
||||
- Cryptography
|
||||
- Secrets Management
|
||||
- Authentication
|
||||
- Authorization (Access Control)
|
||||
- Multi-tenancy Isolation
|
||||
|
||||
Obviously we can skip control families as "not applicable" in the event that the component does not require it. For example,
|
||||
something with the sole purpose of interacting with the local file system may have no meaningful Networking component; this
|
||||
isn't a weakness, it's simply "not applicable."
|
||||
|
||||
For each control family we want to ask:
|
||||
|
||||
- What does the component do for this control?
|
||||
- What sorts of data passes through that control?
|
||||
- for example, a component may have sensitive data (Secrets Management), but that data never leaves the component's storage via Networking
|
||||
- What can attacker do with access to this component?
|
||||
- What's the simplest attack against it?
|
||||
- Are there mitigations that we recommend (i.e. "Always use an interstitial firewall")?
|
||||
- What happens if the component stops working (via DoS or other means)?
|
||||
- Have there been similar vulnerabilities in the past? What were the mitigations?
|
||||
|
||||
# Threat Scenarios
|
||||
|
||||
- An External Attacker without access to the client application
|
||||
- An External Attacker with valid access to the client application
|
||||
- An Internal Attacker with access to cluster
|
||||
- A Malicious Internal User
|
||||
|
||||
## Networking
|
||||
|
||||
## Cryptography
|
||||
|
||||
## Secrets Management
|
||||
|
||||
## Authentication
|
||||
|
||||
## Authorization
|
||||
|
||||
## Multi-tenancy Isolation
|
||||
|
||||
## Summary
|
||||
|
||||
# Recommendations
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Loading…
Reference in New Issue