RE: [virtio-comment] [PATCH v1 1/8] admin: Add theory of operation for device migration

[Parav Pandit <parav@nvidia.com>]

> From: Jason Wang <jasowang@redhat.com>
> Sent: Thursday, October 26, 2023 6:16 AM
> 
> On Wed, Oct 25, 2023 at 3:03âPM Parav Pandit <parav@nvidia.com> wrote:
> >
> >
> > > From: Jason Wang <jasowang@redhat.com>
> > > Sent: Wednesday, October 25, 2023 6:59 AM
> > > > For passthrough PASID assignment vq is not needed.
> > >
> > > How do you know that?
> > Because for passthrough, the hypervisor is not involved in dealing with VQ at
> all.
> 
> Ok, so if I understand correctly, you are saying your design can't work for the
> case of PASID assignment.
>
No. PASID assignment will happen from the guest for its own use and device migration will just work fine because device context will capture this. 
 
> >
> > > There are works ongoing to make vPASID work for the guest like vSVA.
> > > Virtio doesn't differ from other devices.
> > Passthrough do not run like SVA.
> 
> Great, you find another limitation of "passthrough" by yourself.
> 
No. it is not the limitation it is just the way it does not need complex SVA to split the device for unrelated usage.

> > Each passthrough device has PASID from its own space fully managed by the
> guest.
> > Some cpu required vPASID and SIOV is not going this way anmore.
> 
> Then how to migrate? Invent a full set of something else through another giant
> series like this to migrate to the SIOV thing? That's a mess for sure.
>
SIOV will for sure reuse most or all parts of this work, almost entirely as_is.
vPASID is cpu/platform specific things not part of the SIOV devices.

> >
> > >
> > > > If at all it is done, it will be done from the guest by the driver
> > > > using virtio
> > > interface.
> > >
> > > Then you need to trap. Such things couldn't be passed through to guests
> directly.
> > >
> > Only PASID capability is trapped. PASID allocation and usage is directly from
> guest.
> 
> How can you achieve this? Assigning a PAISD to a device is completely
> device(virtio) specific. How can you use a general layer without the knowledge
> of virtio to trap that?
When one wants to map vPASID to pPASID a platform needs to be involved.
When virtio passthrough device is in guest, it has all its PASID accessible.

All these is large deviation from current discussion of this series, so I will keep it short.

> 
> > Regardless it is not relevant to passthrough mode as PASID is yet another
> resource.
> > And for some cpu if it is trapped, it is generic layer, that does not require virtio
> involvement.
> > So virtio interface asking to trap something because generic facility has done
> in not the approach.
> 
> This misses the point of PASID. How to use PASID is totally device specific.
Sure, and how to virtualize vPASID/pPASID is platform specific as single PASID can be used by multiple devices and process.

> 
> >
> > > > Capabilities of #2 is generic across all pci devices, so it will
> > > > be handled by the
> > > HV.
> > > > ATS/PRI cap is also generic manner handled by the HV and PCI device.
> > >
> > > No, ATS/PRI requires the cooperation from the vIOMMU. You can simply
> > > do ATS/PRI passthrough but with an emulated vIOMMU.
> > And that is not the reason for virtio device to build trap+emulation for
> passthrough member devices.
> 
> vIOMMU is emulated by hypervisor with a PRI queue, 
PRI requests arrive on the PF for the VF.

> how can you pass
> through a hardware PRI request to a guest directly without trapping it then?
> What's more, PCIE allows the PRI to be done in a vendor (virtio) specific way, so
> you want to break this rule? Or you want to blacklist ATS/PRI for virtio?
> 
I was aware of only pci-sig way of PRI.
Do you have a reference to the ECN that enables vendor specific way of PRI? I would like to read it.
This will be very good to eliminate IOMMU PRI limitations.
PRI will directly go to the guest driver, and guest would interact with IOMMU to service the paging request through IOMMU APIs.
For PRI in vendor specific way needs a separate discussion. It is not related to live migration.