[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]
Subject: Re: [virtio-comment] About adding a new device type virtio-nvme
å 2023/1/18 10:15, äèä åé:
On Tue, 17 Jan 2023 10:34:09 -0500, Stefan wrote:On Tue, Jan 17, 2023 at 05:41:57PM +0800, äèä wrote:On Tue, 17 Jan 2023 09:32:05 +0100ïDavid wroteïOn 17.01.23 03:04, äèä wrote:virtio-nvmeÂadvantages : 1) live migration 2) support remote storageAt least 1) is an implementation detail in the NVME implementation in the hypervisor. I suspect 2) in a similar way, or is there a fundamental issue with that?One problematic thing about the NVME implementation in QEMU is that it will pin (via vfio) all guest RAM. Could that be avoided using virtio-NVME, or what exactly would be the difference between virtio-nvme and ordinary NVME?In the virtualization scenario where devices are offload to hardwareï NVMEï ---------------------------------------------------------------------------------------------------------------------  _____________________________________________________________________________  | ___________________________________________________________ |  | | _____________________________________________________ | |  | | | | | |  | | | __________________________________ | | |  | | | | ______ | | | | ______  | | | User | | Mem |-----------------------|----|-|----|-----> | |  | | | | |______| SPDK | | | | (gVA) |______|  | | | | (gVA) | | | | | |  | | | |______|___________________________| | | | | |  | | |--GuestOS----------|--------------------------------| | | | |  | | | ______\/__________________________ | | | | |  | | VM | | VFIO | | | | | |  | User | | Kernel |___________ __________________| | | | | |  | | | | vfio-pci | | | | | | | |  | | | | (gIOVA) | | vfio_iommu_type1 | | | | | | Software | | | |______|____|___|__________________| | | | | |  | | |___________________|________________________________| | | | |  | | ___________________|________________________________ | | | |  | | | ______|____ __________________ | | | \/ \/  | | | | \/ | | | | | | ______  | | | | NVME | | vIOMMU | | | | | |  | | | QEMU | Instance | | --|----|-|----|-----> |______|  | | | | (gIOVA) | | (gIOVA-->gPA) | | | | (gPA) | |  | | | |_____|_____| |__________________| | | | | |  | | |__________________|_________________________________| | | | |  | |_______________________|___________________________________| | | |  |---HostOS---------------------------|----------------------------------------| | |  | _______________________|___________________________________ | | |  | | | | | | |  | | | VFIO | | | |  | Kernel |_______________________|_____ _________________________| | | |  | | \/ | | | | | |  | | vfio-pci | | vfio_iommu_type1 | | | |  | | | | | | | | |  | |_______________________|_____|___|_________________________| | | |  |____________________________________|________________________________________| | | -----------------------------------------------|-------------------------------------------------|----|--------------  ____________________________________\/____ _________________________ ___\/___\/___________  | | | | | | | | | | Hardware | | | DMA (gIOVA) --|---|-> IOMMU --|---------|->|______| |  | | |_____________________________| | (gIOVA-->hPA) | (hPA) | Physical Memory |  | DPU | | |_________________________| |_____________________|  | | NVME-of |  | |__________________________________|  | | |  |___________________________|______________| --------------------------------------|------------------------------------------------------------------------------  | TCP (RDMA, and so on)  ______________v__________  | | Remote storage | |  | Network Storage |  | |  |_________________________|---------------------------------------------------------------------------------------------------------------------It is difficult to implement PCIe passthrough live migration.Linux commit 115dcec65f61d53e25e1bed5e380468b30f98b14 ("vfio: Define device migration protocol v2") defines the VFIO migration API and it's implemented by several drivers in the kernel.Yes, this commit supports VFIO live migration, but the feature is a work in progress, recent submission: https://lore.kernel.org/all/20230116141135.12021-10-avihaih@nvidia.com/Can you explain the difficulty of implementing PCIe passthrough live migration in more detail?VFIO live migration requires IOMMU to support dirty page tracking. Currently, no IOMMU device supports this feature. So, VFIO live migration will take a long time. Detailed information referenceïhttps://www.qemu.org/docs/master/devel/vfio-migration.htmlvirtio-nvme: ---------------------------------------------------------------------------------------------------------------------  _____________________________________________________________________________  | ___________________________________________________________ |  | | _____________________________________________________ | |  | | | | | |  | | | __________________________________ | | |  | | | | ______ | | | | ______  | | | User | | Mem |-----------------------|----|-|----|-----> | |  | | | | |______| SPDK | | | | (gVA) |______|  | | | | (gVA) | | | | | |  | | | |______|___________________________| | | | | |  | | |--GuestOS----------|--------------------------------| | | | |  | | | ______v___________________________ | | | | |  | | VM | | VFIO | | | | | |  | User | | Kernel |___________ __________________| | | | | |  | | | | vfio-pci | | | | | | | |  | | | | (gIOVA) | | vfio_iommu_type1 | | | | | | Software | | | |______|____|___|__________________| | | | | |  | | |___________________|________________________________| | | | |  | | ___________________|________________________________ | | | |  | | | ______v____ __________________ | | | v v  | | | |virtio-NVME| | | | | | ______  | | | | Instance | | vIOMMU | | | | | |  | | | QEMU | | | --|----|-|----|-----> |______|  | | | | vhost-vdpa| | (gIOVA-->gPA) | | | | (gPA) | |  | | | |__(gIOVA)__| |__________________| | | | | |  | | |__________________|_________________________________| | | | |  | |_______________________|___________________________________| | | |  |---HostOS---------------------------|----------------------------------------| | |  | _______________________|___________________________________ | | |  | | | | | | |  | | | vDPA | | | |  | Kernel |_______________________v_____ _________________________| | | |  | | | | | | | | |  | | vdpa-device | | | | | | |  | | (Virtual device) | | | | | | |  | |_______________________|_____|___|_________________________| | | |  |____________________________________|________________________________________| | | -----------------------------------------------|-------------------------------------------------|----|--------------  ____________________________________v_____ _________________________ ___v____v____________  | | | | | | | | | | Hardware | | | DMA (gIOVA) --|---|-> IOMMU --|---------|->|______| |  | | |_____________________________| | (gIOVA-->hPA) | (hPA) | Physical Memory |  | DPU | | |_________________________| |_____________________|  | | virtio-nvme-of |  | |__________________________________|  | | |  |___________________________|______________| --------------------------------------|------------------------------------------------------------------------------  | TCP (RDMA, and so on)  ______________v__________  | | Remote storage | |  | Network Storage |  | |  |_________________________|---------------------------------------------------------------------------------------------------------------------Based on the vDPA framework, it supports live migration.The two diagrams are quite similar. Did you want to highlight a difference between the two approaches in the diagram?The biggest difference is the VFIO and vDPA frameworks. The vDPA (virtio data path acceleration) kernel framework is a pillar in productizing the end-to-end vDPA solution and it enables NIC vendors to integrate their vDPA NIC kernel drivers into the framework as part of their productization efforts. Detailed information referenceïhttps://www.redhat.com/en/blog/introduction-vdpa-kernel-framework
Note that vDPA is not solely software concept but also a hardware one. And with vDPA kernel support the device can talks to kernel I/O subsystem via virtio-vDPA bus drivers.
Thanks
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]