RE: [virtio-comment] [PATCH v1 1/8] admin: Add theory of operation for device migration

[Parav Pandit <parav@nvidia.com>]

Hi Lingshan,

> From: Zhu, Lingshan <lingshan.zhu@intel.com>
> Sent: Tuesday, October 10, 2023 2:28 PM
> 
> On 10/10/2023 1:21 AM, Parav Pandit wrote:
> >
> >> From: Michael S. Tsirkin <mst@redhat.com>
> >> Sent: Monday, October 9, 2023 9:50 PM
> >>>>> One or more passthrough PCI VF devices are ubiquitous for virtual
> >>>>> machines usage using generic kernel framework such as vfio [1].
> >>>> Mentioning a specific subsystem in a specific OS may mislead the
> >>>> user to think it can only work in that setup. Let's not do that,
> >>>> virtio is not only used for Linux and VFIO.
> >>> This is just one example on how these commands are useful.
> >>> It can be useful in more ways too in more OSes too.
> >>> I will drop from the patch commit log and keep as information
> >>> purpose in
> >> cover letter.
> >>> Would that work for you?
> >>>
> >>> I donât have any strong opinion to keep it or remove it as most
> >>> stakeholders
> >> has the clear view of requirements now.
> >>> Let me know.
> >> So some people use VFs with VFIO. Hence the module name.  This
> >> sentence by itself seems to have zero value for the spec. Just drop it.
> > Ok. Will drop.
> So why not build your admin vq live migration on our config space solution, get
> out of the troubles, to make your life easier?
> 
Your this question is completely unrelated to this reply or you misunderstood what dropping commit log means.

Dropping link to vfio does not drop the requirement.
I am ok to drop because requirements are clear of passthrough of member device.
Vfio is not a trouble at all.
Admin command is not a trouble either.

The pure technical reason is: all the functionalities proposed cannot be done in any other existing way.
Why? For below reasons.
1. device context, and write records (aka dirty page addresses) is huge which cannot be shared using config registers at scale of 4000 member devices
2. sharing such large context and write addresses in parallel for multiple devices cannot be done using single register file
3. These registers cannot be residing in the VF because VF can undergo FLR, and device reset which must clear these registers
4. When VF does the DMA, all dma occurs in the guest address space, not in hypervisor space; any flr and device reset must stop such dma.
And device reset and flr are controlled by the guest (not mediated by hypervisor).
5. Any PASID to separate out admin vq on the VF does not work for two reasons.
R_1: device flr and device reset must stop all the dmas.
R_2: PASID by most leading vendors is still not mature enough
R_3: One also needs to do inversion to not expose PASID capability of the member PCI device to not expose 

> Actually you don't see any technical problems in our config space proposal,
> right?
In config registers method, for passthrough I clearly see the technical problems (functional and scale) listed above.
Due to which config registers cannot reside on the VF and cannot scale either.