Extend SIG scalability FAQ

sig-scalability/configs-and-limits/faq.md

@@ -25,3 +25,53 @@ As a result, we are consciously not providing any values, because
 one can imagine a cluster that handles thousands of QPS and a cluster
 that is set up the same way may have troubles handling small tens of
 QPS if the requests are fairly expensive.
+
+
+### How do you setup clusters for scalability testing?
+
+We are testing Kubernetes on two levels of scale: 100 nodes and 5000 nodes.
+Given that Kubernetes scalability is a multidimensional problem, we are
+obviously trying to exercise many other dimensions in our tests.
+
+However, all our scalability tests are performed on the cluster with just
+a single large control-plane machine (VM) with all control-plane components
+running that machine.
+The single large machine provides sufficient scalability to let us run
+our load tests on a cluster with 5000 nodes.
+
+The control-plane VM used for testing 5000-node clusters has 64 cores,
+256 GB of RAM and is backed by 200GB SSD persistent disk. Public cloud
+providers typically support even large machines, so as a user you still
+have some slack here.
+
+We are also running a `simulated` clusters ([Kubemark]) where the control
+plane VM is set up the same was as in regular cluster, but many simulated
+nodes are running on a single VM (to simmulate 5000 node clusters, we run
+~80 machines with 60-ish hollow-nodes per machine). However, those are
+currently somewhat auxiliary and release validation is done using a regular
+clusters setup on GCE cloud provider.
+
+[Kubemark]: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-scalability/kubemark-guide.md
+
+
+### Why you are not testing HA cluster (with multiple control-plane instances)?
+
+Honestly, we would like to. But we are lacking capacity to do that.
+The main reason is the tooling to setup cluster - we are still relying on
+kube-up and a bunch of knobs we have there. And kube-up doesn't really
+support HA clusters. However, migration to anything else is a lot of work.
+At the same time we would like to do that consistently for all other jobs
+running on GCE (which all of still rely on kube-up).
+
+We are aware that this is a gap in our testing for several reasons, including:
+* large production clusters generally run at least 3 control plane instances to
+  ensure tolerance to outages of individual instances, zero-downtime upgrade etc.
+  we should be testing what users are doing in real life.
+* etcd is RAFT-based so a cluster of etcd instance have slightly different
+  performance characteristics that a single etcd server
+* distributed systems often show different performance characteristics when the
+  components are separate by a network
+* distributed system may be affected by inconsistency of caches
+
+Noting difference between various configuration is also important as this
+can give indications which optimizations would have the best return-on-investments.

sig-scalability/configs-and-limits/provider-configs.md

@@ -1,23 +1,3 @@
-### Scalability Testing/Analysis Environment and Goals
-
-Project practice is to perform baseline scalability testing and analysis on a large single machine (VM or server) with all control plane processing on that single node. The single large machine provides sufficient scalability to scale to 5000 node density tests. The typical machine for testing at this scale is at the larger end of the VM scale available on public cloud providers, but is by no means the largest available. Large cluster runs are typically run with a node emulator (kubemark), with a set of resources to run kubemark typically requiring ~80 machines to simulate 5000 nodes, i.e. 60-ish hollow-nodes per machine.
-
-The typical control plane server for this testing has 64 cores, at least 128GB of memory, Gig-E network interfaces, and SSD drives. Public cloud instances typically will have more memory than this for this number of cores. 
-
-Several factors contribute to a need to expanded testing beyond the single node baseline.
-
-* Very large cluster operators typically run a minimum of 5 servers in the control plane to ensure that a control plane failure during upgrade is survivable without losing cluster state. We want testing to represent typical configurations at this scale..
-
-* RAFT consensus processing on etcd means that 5-server clusters have different performance characteristics than 1-server clusters.
-
-* Distributed systems often show different performance characteristics when the components are separated by a network versus co-execution on the same server.
-
-* Distributed systems are often affected by cache consistency issues.
-
-An important attribute of Kubernetes is broad support for different infrastructure options. Project experience is that testing on a variety of infrastructure options flushes out timing issues and improves quality. Users of kubernetes also find value in knowing that scale testing has been performed on the infrastructure options they care about.
-
-Scalability testing on configurations that are similar are expected to have similar results, and deviations from the expectation need attention.  Regressions may indicate system issues or undocumented assumptions based on those differences, and should be both explored and documented. Noting differences in various configs and which provide the highest system throughput may also give indications as to which performance optimizations are most interesting.
-
 ### Control Plane Machine Selection
 
 As wide a selection of different infrastructure providers as possible helps the project. Configurations and testing strongly welcomed for providers not currently listed, and the Scalability SIG is engaging with all of the providers listed below.