[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]
Subject: Re: Re: [virtio-comment] About adding a new device type virtio-nvme
On Tue, 17 Jan 2023 09:32:05 +0100ïDavid wroteï >On 17.01.23 03:04, äèä wrote: >> virtio-nvmeÂadvantages : >> 1) live migration >> 2) support remote storage >At least 1) is an implementation detail in the NVME implementation in >the hypervisor. I suspect 2) in a similar way, or is there a fundamental >issue with that? >One problematic thing about the NVME implementation in QEMU is that it >will pin (via vfio) all guest RAM. Could that be avoided using >virtio-NVME, or what exactly would be the difference between virtio-nvme >and ordinary NVME? In the virtualization scenario where devices are offload to hardwareï NVMEï --------------------------------------------------------------------------------------------------------------------- _____________________________________________________________________________ | ___________________________________________________________ | | | _____________________________________________________ | | | | | | | | | | | __________________________________ | | | | | | | ______ | | | | ______ | | | User | | Mem |-----------------------|----|-|----|-----> | | | | | | |______| SPDK | | | | (gVA) |______| | | | | (gVA) | | | | | | | | | |______|___________________________| | | | | | | | |--GuestOS----------|--------------------------------| | | | | | | | ______\/__________________________ | | | | | | | VM | | VFIO | | | | | | | User | | Kernel |___________ __________________| | | | | | | | | | vfio-pci | | | | | | | | | | | | (gIOVA) | | vfio_iommu_type1 | | | | | | Software | | | |______|____|___|__________________| | | | | | | | |___________________|________________________________| | | | | | | ___________________|________________________________ | | | | | | | ______|____ __________________ | | | \/ \/ | | | | \/ | | | | | | ______ | | | | NVME | | vIOMMU | | | | | | | | | QEMU | Instance | | --|----|-|----|-----> |______| | | | | (gIOVA) | | (gIOVA-->gPA) | | | | (gPA) | | | | | |_____|_____| |__________________| | | | | | | | |__________________|_________________________________| | | | | | |_______________________|___________________________________| | | | |---HostOS---------------------------|----------------------------------------| | | | _______________________|___________________________________ | | | | | | | | | | | | | VFIO | | | | | Kernel |_______________________|_____ _________________________| | | | | | \/ | | | | | | | | vfio-pci | | vfio_iommu_type1 | | | | | | | | | | | | | | |_______________________|_____|___|_________________________| | | | |____________________________________|________________________________________| | | -----------------------------------------------|-------------------------------------------------|----|-------------- ____________________________________\/____ _________________________ ___\/___\/___________ | | | | | | | | | | Hardware | | | DMA (gIOVA) --|---|-> IOMMU --|---------|->|______| | | | |_____________________________| | (gIOVA-->hPA) | (hPA) | Physical Memory | | DPU | | |_________________________| |_____________________| | | NVME-of | | |__________________________________| | | | |___________________________|______________| --------------------------------------|------------------------------------------------------------------------------ | TCP ïRDMA, and so onï ______________v__________ | | Remote storage | | | Network Storage | | | |_________________________| --------------------------------------------------------------------------------------------------------------------- It is difficult to implement PCIe passthrough live migration. virtio-nvme: --------------------------------------------------------------------------------------------------------------------- _____________________________________________________________________________ | ___________________________________________________________ | | | _____________________________________________________ | | | | | | | | | | | __________________________________ | | | | | | | ______ | | | | ______ | | | User | | Mem |-----------------------|----|-|----|-----> | | | | | | |______| SPDK | | | | (gVA) |______| | | | | (gVA) | | | | | | | | | |______|___________________________| | | | | | | | |--GuestOS----------|--------------------------------| | | | | | | | ______v___________________________ | | | | | | | VM | | VFIO | | | | | | | User | | Kernel |___________ __________________| | | | | | | | | | vfio-pci | | | | | | | | | | | | (gIOVA) | | vfio_iommu_type1 | | | | | | Software | | | |______|____|___|__________________| | | | | | | | |___________________|________________________________| | | | | | | ___________________|________________________________ | | | | | | | ______v____ __________________ | | | v v | | | |virtio-NVME| | | | | | ______ | | | | Instance | | vIOMMU | | | | | | | | | QEMU | | | --|----|-|----|-----> |______| | | | | vhost-vdpa| | (gIOVA-->gPA) | | | | (gPA) | | | | | |__(gIOVA)__| |__________________| | | | | | | | |__________________|_________________________________| | | | | | |_______________________|___________________________________| | | | |---HostOS---------------------------|----------------------------------------| | | | _______________________|___________________________________ | | | | | | | | | | | | | vDPA | | | | | Kernel |_______________________v_____ _________________________| | | | | | | | | | | | | | | vdpa-device | | | | | | | | | (Virtual device) | | | | | | | | |_______________________|_____|___|_________________________| | | | |____________________________________|________________________________________| | | -----------------------------------------------|-------------------------------------------------|----|-------------- ____________________________________v_____ _________________________ ___v____v____________ | | | | | | | | | | Hardware | | | DMA (gIOVA) --|---|-> IOMMU --|---------|->|______| | | | |_____________________________| | (gIOVA-->hPA) | (hPA) | Physical Memory | | DPU | | |_________________________| |_____________________| | | virtio-nvme-of | | |__________________________________| | | | |___________________________|______________| --------------------------------------|------------------------------------------------------------------------------ | TCP ïRDMA, and so onï ______________v__________ | | Remote storage | | | Network Storage | | | |_________________________| --------------------------------------------------------------------------------------------------------------------- Based on the vDPA framework, it supports live migration. Thanks Leo Hou/houyingle   Â
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]