[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]
Subject: Re: Re: [virtio-comment] About adding a new device type virtio-nvme
On Tue, 17 Jan 2023 09:32:05 +0100ïDavid wroteï
>On 17.01.23 03:04, äèä wrote:
>> virtio-nvmeÂadvantages :
>> 1)Â live migration
>> 2)Â support remote storage
>At least 1) is an implementation detail in the NVME implementation in
>the hypervisor. I suspect 2) in a similar way, or is there a fundamental
>issue with that?
>One problematic thing about the NVME implementation in QEMU is that it
>will pin (via vfio) all guest RAM. Could that be avoided using
>virtio-NVME, or what exactly would be the difference between virtio-nvme
>and ordinary NVME?
In the virtualization scenario where devices are offload to hardwareï
NVMEï
---------------------------------------------------------------------------------------------------------------------
_____________________________________________________________________________
| ___________________________________________________________ |
| | _____________________________________________________ | |
| | | | | |
| | | __________________________________ | | |
| | | | ______ | | | | ______
| | | User | | Mem |-----------------------|----|-|----|-----> | |
| | | | |______| SPDK | | | | (gVA) |______|
| | | | (gVA) | | | | | |
| | | |______|___________________________| | | | | |
| | |--GuestOS----------|--------------------------------| | | | |
| | | ______\/__________________________ | | | | |
| | VM | | VFIO | | | | | |
| User | | Kernel |___________ __________________| | | | | |
| | | | vfio-pci | | | | | | | |
| | | | (gIOVA) | | vfio_iommu_type1 | | | | | |
Software | | | |______|____|___|__________________| | | | | |
| | |___________________|________________________________| | | | |
| | ___________________|________________________________ | | | |
| | | ______|____ __________________ | | | \/ \/
| | | | \/ | | | | | | ______
| | | | NVME | | vIOMMU | | | | | |
| | | QEMU | Instance | | --|----|-|----|-----> |______|
| | | | (gIOVA) | | (gIOVA-->gPA) | | | | (gPA) | |
| | | |_____|_____| |__________________| | | | | |
| | |__________________|_________________________________| | | | |
| |_______________________|___________________________________| | | |
|---HostOS---------------------------|----------------------------------------| | |
| _______________________|___________________________________ | | |
| | | | | | |
| | | VFIO | | | |
| Kernel |_______________________|_____ _________________________| | | |
| | \/ | | | | | |
| | vfio-pci | | vfio_iommu_type1 | | | |
| | | | | | | | |
| |_______________________|_____|___|_________________________| | | |
|____________________________________|________________________________________| | |
-----------------------------------------------|-------------------------------------------------|----|--------------
____________________________________\/____ _________________________ ___\/___\/___________
| | | | | | | | | |
Hardware | | | DMA (gIOVA) --|---|-> IOMMU --|---------|->|______| |
| | |_____________________________| | (gIOVA-->hPA) | (hPA) | Physical Memory |
| DPU | | |_________________________| |_____________________|
| | NVME-of |
| |__________________________________|
| | |
|___________________________|______________|
--------------------------------------|------------------------------------------------------------------------------
| TCP ïRDMA, and so onï
______________v__________
| |
Remote storage | |
| Network Storage |
| |
|_________________________|
---------------------------------------------------------------------------------------------------------------------
It is difficult to implement PCIe passthrough live migration.
virtio-nvme:
---------------------------------------------------------------------------------------------------------------------
_____________________________________________________________________________
| ___________________________________________________________ |
| | _____________________________________________________ | |
| | | | | |
| | | __________________________________ | | |
| | | | ______ | | | | ______
| | | User | | Mem |-----------------------|----|-|----|-----> | |
| | | | |______| SPDK | | | | (gVA) |______|
| | | | (gVA) | | | | | |
| | | |______|___________________________| | | | | |
| | |--GuestOS----------|--------------------------------| | | | |
| | | ______v___________________________ | | | | |
| | VM | | VFIO | | | | | |
| User | | Kernel |___________ __________________| | | | | |
| | | | vfio-pci | | | | | | | |
| | | | (gIOVA) | | vfio_iommu_type1 | | | | | |
Software | | | |______|____|___|__________________| | | | | |
| | |___________________|________________________________| | | | |
| | ___________________|________________________________ | | | |
| | | ______v____ __________________ | | | v v
| | | |virtio-NVME| | | | | | ______
| | | | Instance | | vIOMMU | | | | | |
| | | QEMU | | | --|----|-|----|-----> |______|
| | | | vhost-vdpa| | (gIOVA-->gPA) | | | | (gPA) | |
| | | |__(gIOVA)__| |__________________| | | | | |
| | |__________________|_________________________________| | | | |
| |_______________________|___________________________________| | | |
|---HostOS---------------------------|----------------------------------------| | |
| _______________________|___________________________________ | | |
| | | | | | |
| | | vDPA | | | |
| Kernel |_______________________v_____ _________________________| | | |
| | | | | | | | |
| | vdpa-device | | | | | | |
| | (Virtual device) | | | | | | |
| |_______________________|_____|___|_________________________| | | |
|____________________________________|________________________________________| | |
-----------------------------------------------|-------------------------------------------------|----|--------------
____________________________________v_____ _________________________ ___v____v____________
| | | | | | | | | |
Hardware | | | DMA (gIOVA) --|---|-> IOMMU --|---------|->|______| |
| | |_____________________________| | (gIOVA-->hPA) | (hPA) | Physical Memory |
| DPU | | |_________________________| |_____________________|
| | virtio-nvme-of |
| |__________________________________|
| | |
|___________________________|______________|
--------------------------------------|------------------------------------------------------------------------------
| TCP ïRDMA, and so onï
______________v__________
| |
Remote storage | |
| Network Storage |
| |
|_________________________|
---------------------------------------------------------------------------------------------------------------------
Based on the vDPA framework, it supports live migration.
Thanks
Leo Hou/houyingle
Â
Â
Â
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]