RE: [virtio-comment] [PATCH v1 1/8] admin: Add theory of operation for device migration

[Parav Pandit <parav@nvidia.com>]

> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
> open.org> On Behalf Of Michael S. Tsirkin
> Sent: Thursday, November 16, 2023 12:44 PM
> 
> On Thu, Nov 16, 2023 at 07:02:23AM +0000, Parav Pandit wrote:
> >
> > > From: Michael S. Tsirkin <mst@redhat.com>
> > > Sent: Thursday, November 16, 2023 12:27 PM
> > >
> > > On Thu, Nov 16, 2023 at 06:43:05AM +0000, Parav Pandit wrote:
> > > >
> > > >
> > > > > From: Michael S. Tsirkin <mst@redhat.com>
> > > > > Sent: Thursday, November 16, 2023 12:09 PM
> > > > >
> > > > > On Thu, Nov 16, 2023 at 06:34:23AM +0000, Parav Pandit wrote:
> > > > > >
> > > > > >
> > > > > > > From: Michael S. Tsirkin <mst@redhat.com>
> > > > > > > Sent: Thursday, November 16, 2023 11:53 AM
> > > > > > >
> > > > > > > On Thu, Nov 16, 2023 at 05:28:19AM +0000, Parav Pandit wrote:
> > > > > > > > You continue to want to overload admin commands for dual
> > > > > > > > purpose, does
> > > > > > > not make sense to me.
> > > > > > >
> > > > > > > dual -> as a transport and for migration? why can't they be
> > > > > > > used for this? I was really hoping to cover these two cases
> > > > > > > when I proposed
> > > them.
> > > > > > For following reasons.
> > > > > >
> > > > > > 1. migration needs incremental reads of only changed context
> > > > > > between two reads
> > > > > >
> > > > > > 2. migration writes covers large part of the configurations
> > > > > > not just virtio
> > > > > common config and device config.
> > > > > > Such as configuration occurred through the CVQ. All of these
> > > > > > is not needed
> > > > > when done from guest directly via member's own CVQ.
> > > > > >
> > > > > > For backward compatible SIOV transport, one may need them to
> > > > > > transport
> > > > > without above two properties.
> > > > > >
> > > > > > 3. None of this transport is needed for PFs, VFs and
> > > > > > non-backward compatible
> > > > > SIOVs.
> > > > > > Each device to have its own transport that is not intercepted
> > > > > > by the hypervisor
> > > > > and follow the equivalency principle uniformly for all 3 device types.
> > > > > >
> > > > >
> > > > > To clarify. Above seems to justify why the admin commands for
> > > > > migration must be distinct from admin commands for transport.
> > > > > But I don't see why (e.g. two sets of) admin commands can not be used
> for both.
> > > Do you?
> > > >
> > > > I didn't follow, "used for both".
> > > > Can you please explain?
> > > > Both meaning,
> > > > a. for device migration and
> > > > b. for transporting configuration by owner device on behalf of
> > > > member
> > > device?
> > >
> > > Yes, so one set of commands for migration another for passing config
> > > space accesses. We do in fact have admin commands as transport for
> > > legacy, do we not? And in this model we can have new group types,
> > > e.g. SIOV's subfunction or even a "self" group.
> >
> > This is only need for backward compatible SIOV device.
> > And I am not sure if one should create such or not.
> > In some internal test we see device and platform tend to saturate at various
> levels beyond a certain scale, due to which building backward compatible SIOV
> is not very useful.
> 
> > For non-backward compatible SIOV device, PFs, and VFs all the
> > configurations must be done directly from the driver to device without
> > mediation layers as listed in above #3.
> 
> /facepalm. You keep bringing this up.  There's no must here because all
> hypervisor have some kind of mediation. Whenever you have some solution in
> mind you immediately brand whatever it does not do "not mediation" or "out of
> scope" and whatever it does "mediation". The distinction does not matter.
> 
> 
> > Hence, there is really no point in doing transport VQ for future.
> > Each device doing its runtime configuration using its own transport method
> solves scale and security both uniformly across PF, VF, SIOV.
> 
> Yes, there's a point.
> The point is that low end uses of virtio dwarf the high end scalable things you
> are so preoccupied with - understandably as representative of a hardware
> vendor who wants high margins. This is why we need to keep simple things in
> config space and this means that yes, we will keep expanding config space. And
> this in turn means that if you want to do things over DMA then you need a way
> to access config space over DMA.
> This way needs to use *some kind of command* and maybe that can be admin
> command format.

The low-end uses can always use the comm and interface anyway as they do not care for power, speed, hypercall, scale efficiency and security construct.
And the sw maintenance cost this config work is so negligible compared to the rest of the problems, it is less interesting to do config space anymore.

We have debated this many times, Michael.
I also explained in past that config space over DMA has provisioning problem that the cloud admin does not know if VM is old or new, so it ends up reserving things that may never be used.

Hence, endless growth of config space does not help.

The fact is no device is growing the config space anymore.
I will put all the points in the google doc and share one time, so we don't need to keep debating it over.