Re: Re: [virtio-comment] About adding a new device type virtio-nvme

[Stefan Hajnoczi <stefanha@redhat.com>]

On Wed, Jan 18, 2023 at 10:15:12AM +0800, äèä wrote:
> On Tue, 17 Jan 2023 10:34:09 -0500, Stefan wrote:
> 
> >On Tue, Jan 17, 2023 at 05:41:57PM +0800, äèä wrote:
> >> On Tue, 17 Jan 2023 09:32:05 +0100ïDavid wroteï
> >> >On 17.01.23 03:04, äèä wrote:
> >>
> >>
> >>
> >> >> virtio-nvmeÂadvantages :
> >> >> 1)Â live migration
> >> >> 2)Â support remote storage
> >>
> >>
> >>
> >> >At least 1) is an implementation detail in the NVME implementation in
> >> >the hypervisor. I suspect 2) in a similar way, or is there a fundamental
> >> >issue with that?
> >>
> >>
> >>
> >> >One problematic thing about the NVME implementation in QEMU is that it
> >> >will pin (via vfio) all guest RAM. Could that be avoided using
> >> >virtio-NVME, or what exactly would be the difference between virtio-nvme
> >> >and ordinary NVME?
> >>
> >>
> >>
> >> In the virtualization scenario where devices are offload to hardwareï
> >>
> >>
> >> NVMEï
> >> ---------------------------------------------------------------------------------------------------------------------
> >>ÂÂÂÂÂÂÂÂÂÂÂ _____________________________________________________________________________ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ
> >>ÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂ ___________________________________________________________ÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ
> >>ÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ _____________________________________________________Â |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ
> >>ÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ | |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ
> >>ÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂ __________________________________ÂÂÂÂ | |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ
> >>ÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ ______ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ | |ÂÂÂ |ÂÂÂÂÂÂÂ ______ÂÂÂÂÂÂÂÂÂÂÂÂÂ
> >> | | | User | | Mem |-----------------------|----|-|----|-----> | |ÂÂÂÂÂÂÂÂÂÂÂÂ
> >>ÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂ |______|ÂÂÂÂÂÂÂ SPDKÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ | |ÂÂÂ | (gVA) |______|ÂÂÂÂÂÂÂÂÂÂÂÂ
> >>ÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ (gVA)ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ | |ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂ
> >>ÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |______|___________________________|ÂÂÂ | |ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂ
> >>ÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |--GuestOS----------|--------------------------------| |ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂ
> >>ÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂ ______\/__________________________ÂÂÂÂ | |ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂ
> >>ÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ | VM |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ VFIOÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ | |ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂ
> >>ÂÂÂÂÂÂÂÂÂÂ |ÂÂ UserÂÂÂÂ |ÂÂÂ |ÂÂ KernelÂÂ |___________ÂÂÂÂ __________________|ÂÂÂ | |ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂ
> >> | | | | vfio-pci | | | | | | | |ÂÂÂÂÂÂÂÂÂÂÂÂÂ
> >>ÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |Â (gIOVA)Â |ÂÂ | vfio_iommu_type1 |ÂÂÂ | |ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂ
> >> Software | | | |______|____|___|__________________| | | | | |ÂÂÂÂÂÂÂÂÂÂÂÂÂ
> >>ÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |___________________|________________________________| |ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂ
> >>ÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂ ___________________|________________________________Â |ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂ
> >>ÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂ ______|____ÂÂÂÂ __________________ÂÂÂÂ | |ÂÂÂ |ÂÂÂÂÂÂÂ \/ÂÂ \/ÂÂÂÂÂÂÂÂÂÂÂÂ
> >>ÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂ \/ÂÂ |ÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ | |ÂÂÂ |ÂÂÂÂÂÂÂ ______ÂÂÂÂÂÂÂÂÂÂÂÂÂ
> >>ÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ NVMEÂÂ |ÂÂ |ÂÂÂÂÂ vIOMMUÂÂÂÂÂ |ÂÂÂ | |ÂÂÂ |ÂÂÂÂÂÂ |ÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂ
> >>ÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂ QEMUÂÂÂÂ |Â Instance |ÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ --|----|-|----|-----> |______|ÂÂÂÂÂÂÂÂÂÂÂÂ
> >>ÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |Â (gIOVA)Â |ÂÂ |ÂÂ (gIOVA-->gPA)Â |ÂÂÂ | |ÂÂÂ | (gPA)Â |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂ
> >>ÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |_____|_____|ÂÂ |__________________|ÂÂÂ | |ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂ
> >>ÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |__________________|_________________________________| |ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂ
> >>ÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |_______________________|___________________________________|ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂ
> >>ÂÂÂÂÂÂÂÂÂÂ |---HostOS---------------------------|----------------------------------------|ÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂ
> >>ÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂ _______________________|___________________________________ÂÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂ
> >>ÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂ
> >>ÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ VFIOÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂ
> >>ÂÂÂÂÂÂÂÂÂÂ |ÂÂ KernelÂÂ |_______________________|_____ÂÂÂÂ _________________________|ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂ
> >>ÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ \/ÂÂÂ |ÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂ
> >>ÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂ vfio-pciÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂ |ÂÂÂ vfio_iommu_type1ÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂ
> >>ÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂ |ÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂ
> >>ÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |_______________________|_____|___|_________________________|ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂ
> >>ÂÂÂÂÂÂÂÂÂÂ |____________________________________|________________________________________|ÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂ
> >> -----------------------------------------------|-------------------------------------------------|----|--------------
> >>ÂÂÂÂÂÂÂÂÂÂÂ ____________________________________\/____ÂÂÂÂ _________________________ÂÂÂÂÂÂÂÂÂÂ ___\/___\/___________Â
> >>ÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂ |Â |ÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂ |
> >> Hardware | | | DMA (gIOVA) --|---|-> IOMMU --|---------|->|______| |
> >>ÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂ |ÂÂÂ |_____________________________|ÂÂ |ÂÂÂÂÂ (gIOVA-->hPA)ÂÂÂÂÂ |ÂÂ (hPA) |ÂÂ Physical MemoryÂÂ |
> >>ÂÂÂÂÂÂÂÂÂÂ | DPUÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂ |_________________________|ÂÂÂÂÂÂÂÂ |_____________________|
> >>ÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂ |ÂÂÂÂÂÂÂ NVME-ofÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
> >>ÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂ |__________________________________|
> >>ÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂ |
> >>ÂÂÂÂÂÂÂÂÂÂ |___________________________|______________|
> >> --------------------------------------|------------------------------------------------------------------------------
> >>ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ | TCP (RDMA, and so on)
> >>ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ ______________v__________
> >>ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
> >> Remote storageÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
> >>ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂ Network StorageÂÂÂÂ |
> >>ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
> >>ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |_________________________|
> >>ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ
> >> ---------------------------------------------------------------------------------------------------------------------
> >>
> >>
> >> It is difficult to implement PCIe passthrough live migration.
> 
> 
> 
> Â
> >Linux commit 115dcec65f61d53e25e1bed5e380468b30f98b14 ("vfio: Define
> >device migration protocol v2") defines the VFIO migration API and it's
> >implemented by several drivers in the kernel.
> 
> 
> Yes, this commit supports VFIO live migration, but the feature is a work in progress,
> recent submission: https://lore.kernel.org/all/20230116141135.12021-10-avihaih@nvidia.com/
>  
> 
> >Can you explain the difficulty of implementing PCIe passthrough live
> >migration in more detail?
> 
> VFIO live migration requires IOMMU to support dirty page tracking. Currently, 
> no IOMMU device supports this feature. So, VFIO live migration will take a long time.
> Detailed information referenceïhttps://www.qemu.org/docs/master/devel/vfio-migration.html

Can physical devices can do their own dirty page tracking in the
meantime since they know which pages are being written to?

I have CCed Alex Williamson regarding VFIO.

Stefan

Attachment: signature.asc
Description: PGP signature