Re: [virtio-comment] Re: [PATCH v1 3/8] device-context: Define the device context fields for device migration

["Michael S. Tsirkin" <mst@redhat.com>]

On Wed, Oct 11, 2023 at 10:54:39AM +0000, Parav Pandit wrote:
> 
> > From: Zhu, Lingshan <lingshan.zhu@intel.com>
> > Sent: Wednesday, October 11, 2023 3:38 PM
> > 
> 
> > >> The system admin can choose only passthrough some of the devices for
> > >> nested guests, so passthrough the PF to L1 guest is not a good idea,
> > >> because there can be many devices still work for the host or L1.
> > > Possible. One size does not fit all.
> > > What I expressed is most common scenarios that user care about.
> > don't block existing usecases, don't break the userspace, nested is common.
> Nothing is broken as virtio spec do not have any single construct to support migration.
> If nested is common, can you share the performance number with real virtio device with/without 2 level nesting?

Nested is a use case relevant for virualization. Performance numbers on
current hardware and current market analysis are really beside the
point.

> I frankly donât know how they look like.
> 
> > >
> > >>> In second use case, where one want to bind only one member device to
> > >>> one VM, I think same plumbing can be extended to have another VF, to
> > >>> take
> > >> the role of migration device instead of owner device.
> > >>> I donât see a good way to passthrough and also do in-band migration
> > >>> without
> > >> lot of device specific trap and emulation.
> > >>> I also donât know the cpu performance numbers with 3 levels of
> > >>> nested page
> > >> table translation which to my understanding cannot be accelerated by
> > >> the current cpu.
> > >> host_PA->L1_QEMU_VA->L1_Guest_PA->L1_QEMU_VA->L2_Guest_PA and so
> > on,
> > >> there can be performance overhead, but can be done.
> > >>
> > >> So admin vq migration still don't work for nested, this is surely a blocker.
> > > In specific case of member devices are located at different nest level, it does
> > not.
> > so you got the point, so this series should not be merged.
> > >
> > > Why prevents you have a peer VF do the role of migration driver?
> > > Basically, what I am proposing is, connect two VFs to the L1 guest. One VF is
> > migration driver, one VF is passthrough to L2 guest.
> > > And same scheme works.
> > A peer VF? A management VF? still break the existing usecase. and how do you
> > transfer ownership of L2 VF from PF to L1 VF?
> 
> A peer management VF which services admin command (like PF).
> Ownership of admin command is delegated to the management VF.

That sounds really awkward.

> > >
> > > On the other hand,
> > > Many parts of the cpu subsystem such as PML, page tables do not have N
> > level nesting support either.
> > page tables could be emulated, as showed to you before, just PA to VA, nested
> > PA to nested VA
> > > They all work on top of emulation and pay the price for emulation when
> > nesting is done.
> > > May be that is the first version for virtio too.
> > there are performance overhead, but can be done.
> > >
> > > I frankly feel that nesting support requires industry level eco system support
> > not just in virtio.
> > > Virtio attempting to focus on nested and having nearly same level
> > performance as bare metal seems farfetched.
> > > Maybe I am wrong, as we have not seen such high perf nested env even with
> > sw based device.
> > >
> > > What can be possibly done is,
> > > 1. What admin commands are useful from this series that can be useful for
> > nesting?
> > > 2. What admin commands from current series needs extension for nesting?
> > > 3. What admin commands do not work at all for nesting, and hence, need to
> > have new commands.
> > >
> > > If we can focus on those, maybe we can find common approach that cater to
> > both commands.
> > virtio support nested now, dont let your admin vq LM break this.
> New spec addition is not breaking existing virtio implementation in sw.
> New spec additions of owner and member devices do not apply to non member and non owner devices.
> 
> > >
> > >>> Do you know how does it work for Intel x86_64?
> > >>> Can it do > 2 level of nested page tables? If no, what is the perf
> > >>> characteristics
> > >> to expect?
> > >> of course that can be done, Page table is not a problem, there are
> > >> soft mmu emulation and viommu, through performance overhead.
> > > Due to the performance overheads, I really doubt any cloud operator would
> > use passthrough virtio device for any sensible workload.
> > > But you may know already how nested performance looks like that may be
> > acceptable to users.
> > Many tenants run their nested cluster. Don't break this.
> How new spec addition such as crypto device addition broke net device?
> Or how net vq interrupt moderation breaks existing sw?
> It does not.
> They are driven through their own feature bits and admin command capabilities.
> It does not break any existing deployments.