RE: [virtio-comment] Re: [PATCH 09/11] transport-pci: Describe PCI MMR dev config registers

[Parav Pandit <parav@nvidia.com>]

> From: Jason Wang <jasowang@redhat.com>
> Sent: Monday, April 17, 2023 9:02 PM
> 
> > Isn't the size of BAR and its cap_len exposed by the device?
> 
> Somehow, it's more about how the hypervisor is going to use this, memory
> mapped or trapping. For either case, the hypervisor needs to have virtio
> knowledge in order to finish this.
>
Ok.

> > PCI BAR size of the VF can learn the system page size being different for x86
> (4K) and arm (64K).
> > PCI transport seems to support it.
> 
> Yes this is for SR-IOV but not for other cases. We could invent new facilities for
> sure but the hypervisor can not have this assumption.
>
Yeah, its not assumption.
 

> > > assuming you have two generations of device
> > >
> > > gen1: features x,y
> > > gen2: features x,y,z
> > >
> > > You won't be able to do migration between gen1 and gen2 without
> mediation.
> > Gen1 can easily migrate to gen2, because gen1 has smaller subset than gen2.
> > When gen2 device is composed, feature z is disabled.
> 
> Sure, but this requires a lot of features that do not exist in the spec. E.g it
> assumes the device could be composed on demand which seems to fit the idea
> of transport virtqueue. 
I donât see how transport vq is related.
A device could be composed as PCI VF, PCI SIOV or something else.
Underlying transport will tell how it is composed.
May be underlying transport is a transport VQ, but that is not the only transport.

> So it adds dependencies for migration where a simple
> mediation could be used to solve this without bothering the spec.
>
Mediation of PF and hypervisor is not encouraged anymore as we move towards the CC.
So may be some system will do, but as we have the PCI VFs, there is clear need for non-mediated 1.x devices for such guest VMs.
For legacy kernel mediation is acceptable as there is no CC infrastructure in place on older systems.
 
> >
> > Gen2 to gen1 migration can do software-based migration anyway or through
> mediation.
> > But because gen2 may need to migrate to gen1, hence gen2 to gen2 migration
> also should be done through mediation, doesnât make sense to me.
> 
> It really depends on the design:
> 
> 1) if you want to expose any features that is done by admin virtqueue to a
> guest, mediation is a must (e.g if you want do live migration for
> L1)
> 2) mediation is a must for the idea of transport virtqueue
>
Yes. So both transport options are there.
A PCI VF that doesnât legacy baggage will be just fine without a mediation.
For some reason, if one wants to have mediation, may be there is some option of such new transport.
But such transport cannot be the only transport.
 
> > > So as mentioned in another thread, this is a PCI specific solution:
> > >
> > > 1) feature and config are basic virtio facility
> > > 2) capability is not but specific to PCI transport
> > >
> > So any LM solution will have transport specific checks and virtio level checks.
> 
> So here's the model that is used by Qemu currently:
> 
> 1) Device is emulated, it's the charge of the libvirt to launch Qemu and present
> a stable ABI for guests.
> 2) Datapath doesn't need to care about the hardware details since the
> hardware layout is invisible from guest
> 
> You can see, it's more than sufficient for libvirt to check features/config space,
> it doesn't need to care about the hardware BAR layout. Migration is much
> easier in this way. And we can use transport other than PCI in the guest in this
> case for live migration.
>
Sure works in some use cases.
But it is not the only way to operate it as I explained above where there is requirement to not have mediation for non_legacy interface.

> > Solution needs to cover transport as well as transport is integral part of the
> virtio spec.
> > Each transport layer will implement feature/config/cap in its own way.
> 
> If we can avoid those hardware details to be checked, we should not go for
> that. It's a great ease of the management layer.
Those are mainly RO checks and cheap too. It largely does not involved in the LM or data path flow either.