Expander requests' payloads can be rather heavy under upscale pressure,
as they're compounding all candidates options and unschedulable pods
that could fit each options. Expander responses are a subset of the
requests' payload items.
We're allowing ourself to send arbitrary payload sizes (gRPC
`defaultClientMaxSendMessageSize` is `math.MaxInt32`), but we're prone
to drop expander servers responses to the floor, due to the `4MiB`
`defaultClientMaxReceiveMessageSize`.
The arbitrary 128MiB value is meant to be huge (enough to support eg.
several dozen fat 1MiB pods) but not unlimited. Let me know if you'd
rather see that turned to be a command line flag, or an other value.
Also logging the possible gRPC call errors, as that of great help to
diagnose that kind of issues.
Benjamin Pineau
Evan Sheng