Re: [virtio-comment] [PATCH v1 1/8] admin: Add theory of operation for device migration

["Michael S. Tsirkin" <mst@redhat.com>]

On Thu, Nov 16, 2023 at 07:02:23AM +0000, Parav Pandit wrote:
> 
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Thursday, November 16, 2023 12:27 PM
> > 
> > On Thu, Nov 16, 2023 at 06:43:05AM +0000, Parav Pandit wrote:
> > >
> > >
> > > > From: Michael S. Tsirkin <mst@redhat.com>
> > > > Sent: Thursday, November 16, 2023 12:09 PM
> > > >
> > > > On Thu, Nov 16, 2023 at 06:34:23AM +0000, Parav Pandit wrote:
> > > > >
> > > > >
> > > > > > From: Michael S. Tsirkin <mst@redhat.com>
> > > > > > Sent: Thursday, November 16, 2023 11:53 AM
> > > > > >
> > > > > > On Thu, Nov 16, 2023 at 05:28:19AM +0000, Parav Pandit wrote:
> > > > > > > You continue to want to overload admin commands for dual
> > > > > > > purpose, does
> > > > > > not make sense to me.
> > > > > >
> > > > > > dual -> as a transport and for migration? why can't they be used
> > > > > > for this? I was really hoping to cover these two cases when I proposed
> > them.
> > > > > For following reasons.
> > > > >
> > > > > 1. migration needs incremental reads of only changed context
> > > > > between two reads
> > > > >
> > > > > 2. migration writes covers large part of the configurations not
> > > > > just virtio
> > > > common config and device config.
> > > > > Such as configuration occurred through the CVQ. All of these is
> > > > > not needed
> > > > when done from guest directly via member's own CVQ.
> > > > >
> > > > > For backward compatible SIOV transport, one may need them to
> > > > > transport
> > > > without above two properties.
> > > > >
> > > > > 3. None of this transport is needed for PFs, VFs and non-backward
> > > > > compatible
> > > > SIOVs.
> > > > > Each device to have its own transport that is not intercepted by
> > > > > the hypervisor
> > > > and follow the equivalency principle uniformly for all 3 device types.
> > > > >
> > > >
> > > > To clarify. Above seems to justify why the admin commands for
> > > > migration must be distinct from admin commands for transport. But I
> > > > don't see why (e.g. two sets of) admin commands can not be used for both.
> > Do you?
> > >
> > > I didn't follow, "used for both".
> > > Can you please explain?
> > > Both meaning,
> > > a. for device migration and
> > > b. for transporting configuration by owner device on behalf of member
> > device?
> > 
> > Yes, so one set of commands for migration another for passing config space
> > accesses. We do in fact have admin commands as transport for legacy, do we
> > not? And in this model we can have new group types, e.g. SIOV's subfunction or
> > even a "self" group.
> 
> This is only need for backward compatible SIOV device.
> And I am not sure if one should create such or not.
> In some internal test we see device and platform tend to saturate at various levels beyond a certain scale, due to which building backward compatible SIOV is not very useful.

Can we see some reports of all this performance testing you are doing
behind the scenes? It would be really benefitial since everyone
is just discussing things theoretically and here finally someone
did some experiments. This is a strong point in your favor but not
if you just obliquely refer to "certain scale".

> For non-backward compatible SIOV device, PFs, and VFs all the
> configurations must be done directly from the driver to device without
> mediation layers as listed in above #3.

/facepalm. You keep bringing this up.  There's no must here because all
hypervisor have some kind of mediation. Whenever you have some solution
in mind you immediately brand whatever it does not do "not mediation" or
"out of scope" and whatever it does "mediation". The distinction does
not matter.


> Hence, there is really no point in doing transport VQ for future.
> Each device doing its runtime configuration using its own transport method solves scale and security both uniformly across PF, VF, SIOV.

Yes, there's a point.
The point is that low end uses of virtio dwarf the high end scalable
things you are so preoccupied with - understandably as representative of
a hardware vendor who wants high margins. This is why we need to keep
simple things in config space and this means that yes, we will keep
expanding config space. And this in turn means that if you want
to do things over DMA then you need a way to access config space over DMA.
This way needs to use *some kind of command* and maybe that can be admin
command format.

-- 
MST