We let the addon specify exactly what should be pruned; this approach
is a little more verbose but we're likely generating this
automatically in the common case anyway.
In return for the verbosity, we can likely handle more future cases
and edge cases (for example removing objects that aren't labelled or
are in the wrong namespace).