Re: [virtio-comment] [PATCH RFC] virtio: introduce VIRTIO_F_DEVICE_STOP

[Halil Pasic <pasic@linux.ibm.com>]

On Tue, 22 Dec 2020 20:51:04 +0800
Jason Wang <jasowang@redhat.com> wrote:

> >>   From Qemu point of view, vhost-vDPA is just another type of vhost
> >> backend. Qemu needs to stop virtio (vhost) before it can do migration.
> >> So we require vDPA devices to have the ability of stopping or pausing
> >> its datapath. If the vDPA device is by chance the virtio-PCI device, it
> >> needs an interface for receiving stop/resume command from the driver.
> >>
> >> So the devce stop/resume command was sent from Qemu to vhost-VDPA, then
> >> to vDPA parent which could be a virtio-PCI device in this case.  
> > But QEMU implements the _device_, not the driver, doesn't it?  
> 
> 
> The device is implemented with the co-operation between Qemu and 
> vhost-vDPA. During migration qemu need to stop the virtio-net device, 
> then vhost must be stopped as well.
> 

Sorry for the lazy/stupid question (it's been a while since I last
understood that code), how does this work with vhost based virtio-net?

I don't see ioctl that stops the old classic vhost (no vdpa). I have
a fuzzy remembrance that we 'stop' the notifiers (ioeventfd & irqfd) but
I'm not sure.

And since we are at virtio-net, I ask myself how are 'new requests' and
'requests that is (the device) currently processing' defined for the
receive functionality and the rx queue(s).  Remember the latter ('in
flight requests') need to be all completed. I mean 'in-flight' is pretty
straight-forward for something like virtio-blk, or even for tx but I have
a hard time with rx.

> 
> > And IIUC,
> > vhost-VDPA and the vDPA parent are also on the device side. I feel like
> > I'm missing something essential here.  
> 
> 
> Virtio-PCI driver could be a vDPA parent as well in this case. So we 
> need stop the virtio-pci device.

I used to think about vdpa like a vehicle to make partial virtio support
in HW viable and easy. I.e. when I hear vdpa I have something like this
in mind:
https://www.redhat.com/cms/managed-files/2019-10-02-vdpa-figure5.jpg

Of course the 'physical nic' from the linked picture can be a nic that
supports both the virtio data plane and the virtio control plane (i.e.
what you are referring to as a virtio-pci device). But do we still expect
that device to be connected via vdpa?

The second question is not strictly on topic, but I'm still curious, what
do we plan to do with a lower device that does not support virtio control
plane? In that case we can't go via DEVICE_STOP. Does that mean, there
has to be a vendor specific mechanism equivalent to the DEVICE_STOP
mechanism, or otherwise we are non-migratable?

Regards,
Halil