Re: [virtio-comment] About adding a new device type virtio-nvme

[David Hildenbrand <david@redhat.com>]

On 17.01.23 16:34, Stefan Hajnoczi wrote:
> On Tue, Jan 17, 2023 at 05:41:57PM +0800, äèä wrote:
> > On Tue, 17 Jan 2023 09:32:05 +0100ïDavid wroteï
> > > On 17.01.23 03:04, äèä wrote:
> >
> >
> >
> > > > virtio-nvmeÂadvantages :
> > > > 1)Â live migration
> > > > 2)Â support remote storage
> >
> >
> >
> > > At least 1) is an implementation detail in the NVME implementation in
> > > the hypervisor. I suspect 2) in a similar way, or is there a fundamental
> > > issue with that?
> >
> >
> >
> > > One problematic thing about the NVME implementation in QEMU is that it
> > > will pin (via vfio) all guest RAM. Could that be avoided using
> > > virtio-NVME, or what exactly would be the difference between virtio-nvme
> > > and ordinary NVME?
> >
> >
> >
> > In the virtualization scenario where devices are offload to hardwareï
> >
> >
> > NVMEï
> > ---------------------------------------------------------------------------------------------------------------------
> >             _____________________________________________________________________________
> >            |             ___________________________________________________________     |
> >            |            |    _____________________________________________________  |    |
> >            |            |    |                                                    | |    |
> >            |            |    |             __________________________________     | |    |
> >            |            |    |            |    ______                        |    | |    |        ______
> >            |            |    |   User     |   | Mem  |-----------------------|----|-|----|-----> |      |
> >            |            |    |            |   |______|        SPDK           |    | |    | (gVA) |______|
> >            |            |    |            |    (gVA)                         |    | |    |        |    |
> >            |            |    |            |______|___________________________|    | |    |        |    |
> >            |            |    |--GuestOS----------|--------------------------------| |    |        |    |
> >            |            |    |             ______\/__________________________     | |    |        |    |
> >            |            | VM |            |                VFIO              |    | |    |        |    |
> >            |   User     |    |   Kernel   |___________     __________________|    | |    |        |    |
> >            |            |    |            | vfio-pci  |   |                  |    | |    |        |    |
> >            |            |    |            |  (gIOVA)  |   | vfio_iommu_type1 |    | |    |        |    |
> > Software  |            |    |            |______|____|___|__________________|    | |    |        |    |
> >            |            |    |___________________|________________________________| |    |        |    |
> >            |            |     ___________________|________________________________  |    |        |    |
> >            |            |    |             ______|____     __________________     | |    |        \/   \/
> >            |            |    |            |      \/   |   |                  |    | |    |        ______
> >            |            |    |            |    NVME   |   |      vIOMMU      |    | |    |       |      |
> >            |            |    |   QEMU     |  Instance |   |                --|----|-|----|-----> |______|
> >            |            |    |            |  (gIOVA)  |   |   (gIOVA-->gPA)  |    | |    | (gPA)  |    |
> >            |            |    |            |_____|_____|   |__________________|    | |    |        |    |
> >            |            |    |__________________|_________________________________| |    |        |    |
> >            |            |_______________________|___________________________________|    |        |    |
> >            |---HostOS---------------------------|----------------------------------------|        |    |
> >            |             _______________________|___________________________________     |        |    |
> >            |            |                       |                                   |    |        |    |
> >            |            |                       |    VFIO                           |    |        |    |
> >            |   Kernel   |_______________________|_____     _________________________|    |        |    |
> >            |            |                       \/    |   |                         |    |        |    |
> >            |            |        vfio-pci             |   |    vfio_iommu_type1     |    |        |    |
> >            |            |                       |     |   |                         |    |        |    |
> >            |            |_______________________|_____|___|_________________________|    |        |    |
> >            |____________________________________|________________________________________|        |    |
> > -----------------------------------------------|-------------------------------------------------|----|--------------
> >             ____________________________________\/____     _________________________           ___\/___\/___________
> >            |       |    |                             |   |                         |         |  |      |           |
> > Hardware  |       |    |          DMA     (gIOVA)  --|---|->        IOMMU        --|---------|->|______|           |
> >            |       |    |_____________________________|   |      (gIOVA-->hPA)      |   (hPA) |   Physical Memory   |
> >            | DPU   |                                  |   |_________________________|         |_____________________|
> >            |       |        NVME-of                   |
> >            |       |__________________________________|
> >            |                           |              |
> >            |___________________________|______________|
> > --------------------------------------|------------------------------------------------------------------------------
> >                                        | TCP (RDMA, and so on)
> >                          ______________v__________
> >                         |                         |
> > Remote storage         |                         |
> >                         |     Network Storage     |
> >                         |                         |
> >                         |_________________________|
> >                         
> > ---------------------------------------------------------------------------------------------------------------------
> >
> >
> > It is difficult to implement PCIe passthrough live migration.
>
> Linux commit 115dcec65f61d53e25e1bed5e380468b30f98b14 ("vfio: Define
> device migration protocol v2") defines the VFIO migration API and it's
> implemented by several drivers in the kernel.
>
> Can you explain the difficulty of implementing PCIe passthrough live
> migration in more detail?
>
> >   
> > virtio-nvme:
> > ---------------------------------------------------------------------------------------------------------------------
> >             _____________________________________________________________________________
> >            |             ___________________________________________________________     |
> >            |            |    _____________________________________________________  |    |
> >            |            |    |                                                    | |    |
> >            |            |    |             __________________________________     | |    |
> >            |            |    |            |    ______                        |    | |    |        ______
> >            |            |    |   User     |   | Mem  |-----------------------|----|-|----|-----> |      |
> >            |            |    |            |   |______|        SPDK           |    | |    | (gVA) |______|
> >            |            |    |            |    (gVA)                         |    | |    |        |    |
> >            |            |    |            |______|___________________________|    | |    |        |    |
> >            |            |    |--GuestOS----------|--------------------------------| |    |        |    |
> >            |            |    |             ______v___________________________     | |    |        |    |
> >            |            | VM |            |                VFIO              |    | |    |        |    |
> >            |   User     |    |   Kernel   |___________     __________________|    | |    |        |    |
> >            |            |    |            | vfio-pci  |   |                  |    | |    |        |    |
> >            |            |    |            |  (gIOVA)  |   | vfio_iommu_type1 |    | |    |        |    |
> > Software  |            |    |            |______|____|___|__________________|    | |    |        |    |
> >            |            |    |___________________|________________________________| |    |        |    |
> >            |            |     ___________________|________________________________  |    |        |    |
> >            |            |    |             ______v____     __________________     | |    |        v    v
> >            |            |    |            |virtio-NVME|   |                  |    | |    |        ______
> >            |            |    |            |  Instance |   |      vIOMMU      |    | |    |       |      |
> >            |            |    |   QEMU     |           |   |                --|----|-|----|-----> |______|
> >            |            |    |            | vhost-vdpa|   |   (gIOVA-->gPA)  |    | |    | (gPA)  |    |
> >            |            |    |            |__(gIOVA)__|   |__________________|    | |    |        |    |
> >            |            |    |__________________|_________________________________| |    |        |    |
> >            |            |_______________________|___________________________________|    |        |    |
> >            |---HostOS---------------------------|----------------------------------------|        |    |
> >            |             _______________________|___________________________________     |        |    |
> >            |            |                       |                                   |    |        |    |
> >            |            |                       |    vDPA                           |    |        |    |
> >            |   Kernel   |_______________________v_____     _________________________|    |        |    |
> >            |            |                       |     |   |                         |    |        |    |
> >            |            |   vdpa-device         |     |   |                         |    |        |    |
> >            |            | (Virtual device)      |     |   |                         |    |        |    |
> >            |            |_______________________|_____|___|_________________________|    |        |    |
> >            |____________________________________|________________________________________|        |    |
> > -----------------------------------------------|-------------------------------------------------|----|--------------
> >             ____________________________________v_____     _________________________           ___v____v____________
> >            |       |    |                             |   |                         |         |  |      |           |
> > Hardware  |       |    |          DMA     (gIOVA)  --|---|->        IOMMU        --|---------|->|______|           |
> >            |       |    |_____________________________|   |      (gIOVA-->hPA)      |   (hPA) |   Physical Memory   |
> >            |  DPU  |                                  |   |_________________________|         |_____________________|
> >            |       |        virtio-nvme-of            |
> >            |       |__________________________________|
> >            |                           |              |
> >            |___________________________|______________|
> > --------------------------------------|------------------------------------------------------------------------------
> >                                        | TCP (RDMA, and so on)
> >                          ______________v__________
> >                         |                         |
> > Remote storage         |                         |
> >                         |     Network Storage     |
> >                         |                         |
> >                         |_________________________|
> >                         
> > ---------------------------------------------------------------------------------------------------------------------
> > Based on the vDPA framework, it supports live migration.
>
> The two diagrams are quite similar. Did you want to highlight a
> difference between the two approaches in the diagram?

I also wondered why virtio-nvme is exactly needed, and why one couldn't 
write a different "backend" for an ordinary NVME device, that talks to 
the vdpa-device.

--
Thanks,

David / dhildenb