[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]
Subject: Re: [virtio-comment] Live Migration of Virtio Virtual Function
On Wed, Aug 18, 2021 at 5:16 PM Max Gurtovoy <mgurtovoy@nvidia.com> wrote: > > > On 8/17/2021 12:44 PM, Jason Wang wrote: > > On Tue, Aug 17, 2021 at 5:11 PM Max Gurtovoy <mgurtovoy@nvidia.com> wrote: > >> > >> On 8/17/2021 11:51 AM, Jason Wang wrote: > >>> å 2021/8/12 äå8:08, Max Gurtovoy åé: > >>>> Hi all, > >>>> > >>>> Live migration is one of the most important features of > >>>> virtualization and virtio devices are oftenly found in virtual > >>>> environments. > >>>> > >>>> The migration process is managed by a migration SW that is running on > >>>> the hypervisor and the VM is not aware of the process at all. > >>>> > >>>> Unlike the vDPA case, a real pci Virtual Function state resides in > >>>> the HW. > >>>> > >>> vDPA doesn't prevent you from having HW states. Actually from the view > >>> of the VMM(Qemu), it doesn't care whether or not a state is stored in > >>> the software or hardware. A well designed VMM should be able to hide > >>> the virtio device implementation from the migration layer, that is how > >>> Qemu is wrote who doesn't care about whether or not it's a software > >>> virtio/vDPA device or not. > >>> > >>> > >>>> In our vision, in order to fulfil the Live migration requirements for > >>>> virtual functions, each physical function device must implement > >>>> migration operations. Using these operations, it will be able to > >>>> master the migration process for the virtual function devices. Each > >>>> capable physical function device has a supervisor permissions to > >>>> change the virtual function operational states, save/restore its > >>>> internal state and start/stop dirty pages tracking. > >>>> > >>> For "supervisor permissions", is this from the software point of view? > >>> Maybe it's better to give an example for this. > >> A permission to a PF device for quiesce and freeze a VF device for example. > > Note that for safety, VMM (e.g Qemu) is usually running without any privileges. > > You're mixing layers here. > > QEMU is not involved here. It's only sending IOCTLs to migration driver. > The migration driver will control the migration process of the VF using > the PF communication channel. So who will be granted the "permission" you mentioned here? > > > > > >>> > >>>> An example of this approach can be seen in the way NVIDIA performs > >>>> live migration of a ConnectX NIC function: > >>>> > >>>> https://github.com/jgunthorpe/linux/commits/mlx5_vfio_pci > >>>> <https://github.com/jgunthorpe/linux/commits/mlx5_vfio_pci> > >>>> > >>>> NVIDIAs SNAP technology enables hardware-accelerated software defined > >>>> PCIe devices. virtio-blk/virtio-net/virtio-fs SNAP used for storage > >>>> and networking solutions. The host OS/hypervisor uses its standard > >>>> drivers that are implemented according to a well-known VIRTIO > >>>> specifications. > >>>> > >>>> In order to implement Live Migration for these virtual function > >>>> devices, that use a standard drivers as mentioned, the specification > >>>> should define how HW vendor should build their devices and for SW > >>>> developers to adjust the drivers. > >>>> > >>>> This will enable specification compliant vendor agnostic solution. > >>>> > >>>> This is exactly how we built the migration driver for ConnectX > >>>> (internal HW design doc) and I guess that this is the way other > >>>> vendors work. > >>>> > >>>> For that, I would like to know if the approach of âPF that controls > >>>> the VF live migration processâ is acceptable by the VIRTIO technical > >>>> group ? > >>>> > >>> I'm not sure but I think it's better to start from the general > >>> facility for all transports, then develop features for a specific > >>> transport. > >> a general facility for all transports can be a generic admin queue ? > > It could be a virtqueue or a transport specific method (pcie capability). > > No. You said a general facility for all transports. For general facility, I mean the chapter 2 of the spec which is general " 2 Basic Facilities of a Virtio Device " > > Transport specific is not general. The transport is in charge of implementing the interface for those facilities. > > > > > E.g we can define what needs to be migrated for the virtio-blk first > > (the device state). Then we can define the interface to get and set > > those states via admin virtqueue. Such decoupling may ease the future > > development of the transport specific migration interface. > > I asked a simple question here. > > Lets stick to this. I answered this question. The virtqueue could be one of the approaches. And it's your responsibility to convince the community about that approach. Having an example may help people to understand your proposal. > > I'm not referring to internal state definitions. Without an example, how do we know if it can work well? > > Can you please not change the subject of my initial intent in the email ? Did I? Basically, I'm asking how a virtio-blk can be migrated with your proposal. Thanks > > Thanks. > > > > > > Thanks > > > >> > >>> Thanks > >>> > >>> > >>>> Cheers, > >>>> > >>>> -Max. > >>>> > >> This publicly archived list offers a means to provide input to the > >> OASIS Virtual I/O Device (VIRTIO) TC. > >> > >> In order to verify user consent to the Feedback License terms and > >> to minimize spam in the list archive, subscription is required > >> before posting. > >> > >> Subscribe: virtio-comment-subscribe@lists.oasis-open.org > >> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > >> List help: virtio-comment-help@lists.oasis-open.org > >> List archive: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.oasis-open.org%2Farchives%2Fvirtio-comment%2F&data=04%7C01%7Cmgurtovoy%40nvidia.com%7C0c1898923999420201b408d961639dee%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637647902764977086%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=ZvEbfxoQhPnFyy1m%2BM5%2FyD0UW%2FJ7YZnOliFepRsoU30%3D&reserved=0 > >> Feedback License: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fwho%2Fipr%2Ffeedback_license.pdf&data=04%7C01%7Cmgurtovoy%40nvidia.com%7C0c1898923999420201b408d961639dee%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637647902764977086%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Xsf04HBGeQb4qOwgAfUQucHsizBHV5Vj2WOJVc0AYTs%3D&reserved=0 > >> List Guidelines: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fpolicies-guidelines%2Fmailing-lists&data=04%7C01%7Cmgurtovoy%40nvidia.com%7C0c1898923999420201b408d961639dee%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637647902764977086%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=GUVBMKmEmZxbhacMofMW9OitOgMscoIOOvS%2BkyPTg1Y%3D&reserved=0 > >> Committee: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fcommittees%2Fvirtio%2F&data=04%7C01%7Cmgurtovoy%40nvidia.com%7C0c1898923999420201b408d961639dee%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637647902764977086%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=OTdKqw11N6FxaSNhxBlmLSheYYqnHGVGyfgyuoAzxOU%3D&reserved=0 > >> Join OASIS: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fjoin%2F&data=04%7C01%7Cmgurtovoy%40nvidia.com%7C0c1898923999420201b408d961639dee%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637647902764977086%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=e80lScamJ3Sn5NqsiDBZ3tpc8oHl%2F0dVOq39pGHH4v8%3D&reserved=0 > >> >
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]