Re: [virtio-comment] Live Migration of Virtio Virtual Function

[Jason Wang <jasowang@redhat.com>]

在 2021/8/19 下午10:58, Michael S. Tsirkin 写道:
> On Thu, Aug 19, 2021 at 10:44:46AM +0800, Jason Wang wrote:
> > > The PF device will have an option to quiesce/freeze the VF device.
> >
> > Is such design a must? If no, why not simply introduce those functions in
> > the VF?
> Many IOMMUs only support protections at the function level.
> Thus we need ability to have one device (e.g. a PF)
> to control migration of another (e.g. a VF).


So as discussed previously, the only possible "advantage" is that the 
DMA is isolated.


> This is because allowing VF to access hypervisor memory used for
> migration is not a good idea.
> For IOMMUs that support subfunctions, these "devices" could be
> subfunctions.
>
> The only alternative is to keep things in device memory which
> does not need an IOMMU.
> I guess we'd end up with something like a VQ in device memory which might
> be tricky from multiple points of view, but yes, this could be
> useful and people did ask for such a capability in the past.


I assume the spec already support this. We probably need some 
clarification at the transport layer. But it's as simple as setting MMIO 
are as virtqueue address?

Except for the dirty bit tracking, we don't have bulk data that needs to 
be transferred during migration. So a virtqueue is not must even in this 
case.


> > If yes, what's the reason for making virtio different (e.g VCPU live
> > migration is not designed like that)?
> I think the main difference is we need PF's help for memory
> tracking for pre-copy migration anyway.


Such kind of memory tracking is not a must. KVM uses software assisted 
technologies (write protection) and it works very well. For virtio, 
technology like shadow virtqueue has been used by DPDK and prototyped by 
Eugenio.

Even if we want to go with hardware technology, we have many 
alternatives (as we've discussed in the past):

1) IOMMU dirty bit (E.g modern IOMMU have EA bit for logging external 
device write)
2) Write protection via IOMMU or device MMU
3) Address space ID for isolating DMAs

Using physical function is sub-optimal that all of the above since:

1) limited to a specific transport or implementation and it doesn't work 
for device or transport without PF
2) the virtio level function is not self contained, this makes any 
feature that ties to PF impossible to be used in the nested layer
3) more complicated than leveraging the existing facilities provided by 
the platform or transport

Consider (P)ASID will be ready very soon, workaround the platform 
limitation via PF is not a good idea for me. Especially consider it's 
not a must and we had already prototype the software assisted technology.


>   Might as well integrate
> the rest of state in the same channel.


That's another question. I think for the function that is a must for 
doing live migration, introducing them in the function itself is the 
most natural way since we did all the other facilities there. This ease 
the function that can be used in the nested layer.

And using the channel in the PF is not coming for free. It requires 
synchronization in the software or even QOS.

Or we can just separate the dirty page tracking into PF (but need to 
define them as basic facility for future extension).


> Another answer is that CPUs trivially switch between
> functions by switching the active page tables. For PCI DMA
> it is all much trickier sine the page tables can be separate
> from the device, and assumed to be mostly static.


I don't see much different, the page table is also separated from the 
CPU. If the device supports state save and restore we can scheduling the 
multiple VMs/VCPUs on the same device.


> So if you want to create something like the VMCS then
> again you either need some help from another device or
> put it in device memory.


For CPU virtualization, the states could be saved and restored via MSRs. 
For virtio, accessing them via registers is also possible and much more 
simple.

Thanks


>