RE: [virtio-comment] Re: [PATCH v1 3/8] device-context: Define the device context fields for device migration

[Parav Pandit <parav@nvidia.com>]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Thursday, October 12, 2023 4:15 PM

[..]
> > Please donât use the verb "break".
> > Your proposal is the first of its kind that supports migrating nested device.
> > This is why new patches of config register or admin command does not break
> anything existing.
> 
> Wording aside, new features should support as wide a variety of configs as
> possible, if some config is not supported there should be a very good reason.
> 
I agree, if possible, we should define one solution that can work for nested and non-nested case at same level of performance.
I just donât think that it is possible for PCI devices.
Additionally, the eco-system for N level nesting is also not in place for page tables, PML etc to my knowledge.
And I may be totally wrong.

[..]
> > > > It does not break any existing deployments.
> > > we are talking about nested, don't break nested
> > Virtio spec for nested is not defined yet. Hence nothing is broken. Please avoid
> using the verb, _break_.
> 
> Well people are passing virtio devices through to nested guests.
> Ideally such configs should, somehow, support nested hypervisors migrating
> nested guests. 
I think nesting needs some kind of mediation and passthrough needs to avoid it.
So best I can think of somehow the admin commands of this proposal if they can work on the AQ of the member device, it may work.
And if Lingshan can help to extend these commands it will be really nice.

Or the second idea you proposed of dummy PF, can seamlessly work with admin commands too.

> Considering e.g. write tracking needs decent performance for
> live migration to deserve the name, I doubt pulling data across PCIe with
> synchronous MMIO operations with no pipelining will work well enough.
Right. Even Intel PML does not support nesting as per my last read.