Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

[Jason Wang <jasowang@redhat.com>]

å 2021/7/20 äå9:09, Max Gurtovoy åé:
> On 7/20/2021 3:57 PM, Stefan Hajnoczi wrote:
> > On Tue, Jul 20, 2021 at 03:27:00PM +0300, Max Gurtovoy wrote:
> > > On 7/20/2021 6:02 AM, Jason Wang wrote:
> > > > å 2021/7/19 äå8:43, Stefan Hajnoczi åé:
> > > > > On Fri, Jul 16, 2021 at 10:03:17AM +0800, Jason Wang wrote:
> > > > > > å 2021/7/15 äå6:01, Stefan Hajnoczi åé:
> > > > > > > On Thu, Jul 15, 2021 at 09:35:13AM +0800, Jason Wang wrote:
> > > > > > > > å 2021/7/14 äå11:07, Stefan Hajnoczi åé:
> > > > > > > > > On Wed, Jul 14, 2021 at 06:29:28PM +0800, Jason Wang wrote:
> > > > > > > > > > å 2021/7/14 äå5:53, Stefan Hajnoczi åé:
> > > > > > > > > > > On Tue, Jul 13, 2021 at 08:16:35PM +0800, Jason Wang wrote:
> > > > > > > > > > > > å 2021/7/13 äå6:00, Stefan Hajnoczi åé:
> > > > > > > > > > > > > On Tue, Jul 13, 2021 at 11:27:03AM +0800, Jason Wang wrote:
> > > > > > > > > > > > > > å 2021/7/12 äå5:57, Stefan Hajnoczi åé:
> > > > > > > > > > > > > > > On Mon, Jul 12, 2021 at 12:00:39PM +0800, Jason Wang wrote:
> > > > > > > > > > > > > > > > å 2021/7/11 äå4:36, Michael S. Tsirkin åé:
> > > > > > > > > > > > > > > > > On Fri, Jul 09, 2021
> > > > > > > > > > > > > > > > > at 07:23:33PM +0200,
> > > > > > > > > > > > > > > > > Eugenio Perez Martin
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > That's
> > > > > > > > > basically the difference between the vhost/vDPA's
> > > > > > > > > selective passthrough
> > > > > > > > > approach and VFIO's full passthrough approach.
> > > > > > > > We can't do VFIO full pasthrough for migration anyway,
> > > > > > > > some kind of mdev is
> > > > > > > > required but it's duplicated with the current vp_vdpa driver.
> > > > > > > I'm not sure that's true. Generic VFIO PCI migration can 
> > > > > > > probably be
> > > > > > > achieved without mdev:
> > > > > > > 1. Define a migration PCI Capability that indicates support for
> > > > > > > ÂÂÂÂÂ VFIO_REGION_TYPE_MIGRATION. This allows the PCI device
> > > > > > > to implement
> > > > > > > ÂÂÂÂÂ the migration interface in hardware instead of an mdev 
> > > > > > > driver.
> > > > > > So I think it still depend on the driver to implement migrate
> > > > > > state which is
> > > > > > vendor specific.
> > > > > The current VFIO migration interface depends on a device-specific
> > > > > software mdev driver but here I'm showing that the physical device 
> > > > > can
> > > > > implement the migration interface so that no device-specific 
> > > > > driver code
> > > > > is needed.
> > > >
> > > > This is not what I read from the patch:
> > > >
> > > > ÂÂ* device_state: (read/write)
> > > > ÂÂ*ÂÂÂÂÂ - The user application writes to this field to inform the 
> > > > vendor
> > > > driver
> > > > ÂÂ*ÂÂÂÂÂÂÂ about the device state to be transitioned to.
> > > > ÂÂ*ÂÂÂÂÂ - The vendor driver should take the necessary actions to 
> > > > change
> > > > the
> > > > ÂÂ*ÂÂÂÂÂÂÂ device state. After successful transition to a given 
> > > > state, the
> > > > ÂÂ*ÂÂÂÂÂÂÂ vendor driver should return success on write(device_state,
> > > > state)
> > > > ÂÂ*ÂÂÂÂÂÂÂ system call. If the device state transition fails, the 
> > > > vendor
> > > > driver
> > > > ÂÂ*ÂÂÂÂÂÂÂ should return an appropriate -errno for the fault 
> > > > condition.
> > > >
> > > > Vendor driver need to mediate between the uAPI and the actual device.
> > > We're building an infrastructure for VFIO PCI devices in the last few
> > > months.
> > >
> > > It should be merged hopefully to kernel 5.15.
> > Do you have links to patch series or a brief description of the VFIO API
> > features that are on the roadmap?
>
> we devided it to few patchsets .
>
> The entire series can be found at:
>
> https://github.com/jgunthorpe/linux/commits/mlx5_vfio_pci
>
> We'll first add support for mlx5 devices suspend/resume (ConnectX-6 
> and above).
>
> The driver is ready in the series above.
>
> The next step is to add Live migration for virtio. Hopefully we will 
> be able to agree on the Virtio PCI in the near future.


So I still think for vitio pci the migration should be done via vp_vdpa 
driver instead of using vfio/mdev.

Using vDPA have too much advantages (we had a lot of discussion in the 
past) via present a virtio device instead of a transport specific one:

1) software/management stack is ready
2) transport independent, micro VM ready
3) migration compatibility could be maintained
4) management API ready
5) fail-over or multi-path in the future

etc.


> > > > > > Note that it's just an uAPI definition not something defined in 
> > > > > > the PCI
> > > > > > spec.
> > > > > Yes, that's why I mentioned Changpeng Liu's idea to turn the uAPI 
> > > > > into a
> > > > > standard PCI Capability to eliminate the need for device-specific
> > > > > drivers.
> > > >
> > > > Ok.
> > > >
> > > >
> > > > > > Out of curiosity, the patch is merged without any real users in
> > > > > > the Linux.
> > > > > > This is very bad since we lose the change to audit the whole design.
> > > > > I agree. It would have helped to have a complete vision for how live
> > > > > migration should work along with demos. I don't see any migration 
> > > > > code
> > > > > in samples/vfio-mdev/ :(.
> > > >
> > > > Right.
> > > Creating a standard is not related to Linux nor VFIO.
> > >
> > > With the proposal that I've sent, we can develop a migration driver and
> > > virtio device that will support it (NVIDIA virtio-blk SNAP device).
> > >
> > > And you can build live migration support in virtio_vdpa driver (if VDPA
> > > migration protocol will be implemented).
> > I guess "VDPA migration protocol" is not referring to Jason's proposal
> > but instead to a new vDPA interface that adds the migration interface
> > from you proposal to vDPA?
>
> I mean a parallel protocol to VFIO.


Actually, the first step is to define the state and its API in the spec 
as both of us tries to achieve and make it independent to any kind of 
implementation (VFIO or vDPA).

Thanks


> > Stefan