[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]
Subject: Re: Re: [virtio-comment] About adding a new device type virtio-nvme
On Tue, 17 Jan 2023 16:47:23 +0100, David wrote:
>On 17.01.23 16:34, Stefan Hajnoczi wrote:
>> On Tue, Jan 17, 2023 at 05:41:57PM +0800, äèä wrote:
>>> On Tue, 17 Jan 2023 09:32:05 +0100ïDavid wroteï
>>>> On 17.01.23 03:04, äèä wrote:
>>>
>>>
>>>
>>>>> virtio-nvmeÂadvantages :
>>>>> 1)Â live migration
>>>>> 2)Â support remote storage
>>>
>>>
>>>
>>>> At least 1) is an implementation detail in the NVME implementation in
>>>> the hypervisor. I suspect 2) in a similar way, or is there a fundamental
>>>> issue with that?
>>>
>>>
>>>
>>>> One problematic thing about the NVME implementation in QEMU is that it
>>>> will pin (via vfio) all guest RAM. Could that be avoided using
>>>> virtio-NVME, or what exactly would be the difference between virtio-nvme
>>>> and ordinary NVME?
>>>
>>>
>>>
>>> In the virtualization scenario where devices are offload to hardwareï
>>>
>>>
>>> NVMEï
>>> ---------------------------------------------------------------------------------------------------------------------
>>>ÂÂÂÂÂÂÂÂÂÂÂÂ _____________________________________________________________________________
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂ ___________________________________________________________ÂÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ _____________________________________________________Â |ÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ | |ÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂ __________________________________ÂÂÂÂ | |ÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ ______ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ | |ÂÂÂ |ÂÂÂÂÂÂÂ ______
>>> | | | User | | Mem |-----------------------|----|-|----|-----> | |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂ |______|ÂÂÂÂÂÂÂ SPDKÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ | |ÂÂÂ | (gVA) |______|
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ (gVA)ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ | |ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |______|___________________________|ÂÂÂ | |ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |--GuestOS----------|--------------------------------| |ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂ ______\/__________________________ÂÂÂÂ | |ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ | VM |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ VFIOÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ | |ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂ UserÂÂÂÂ |ÂÂÂ |ÂÂ KernelÂÂ |___________ÂÂÂÂ __________________|ÂÂÂ | |ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |
>>> | | | | vfio-pci | | | | | | | |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |Â (gIOVA)Â |ÂÂ | vfio_iommu_type1 |ÂÂÂ | |ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |
>>> Software | | | |______|____|___|__________________| | | | | |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |___________________|________________________________| |ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂ ___________________|________________________________Â |ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂ ______|____ÂÂÂÂ __________________ÂÂÂÂ | |ÂÂÂ |ÂÂÂÂÂÂÂ \/ÂÂ \/
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂ \/ÂÂ |ÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ | |ÂÂÂ |ÂÂÂÂÂÂÂ ______
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ NVMEÂÂ |ÂÂ |ÂÂÂÂÂ vIOMMUÂÂÂÂÂ |ÂÂÂ | |ÂÂÂ |ÂÂÂÂÂÂ |ÂÂÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂ QEMUÂÂÂÂ |Â Instance |ÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ --|----|-|----|-----> |______|
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |Â (gIOVA)Â |ÂÂ |ÂÂ (gIOVA-->gPA)Â |ÂÂÂ | |ÂÂÂ | (gPA)Â |ÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |_____|_____|ÂÂ |__________________|ÂÂÂ | |ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |__________________|_________________________________| |ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |_______________________|___________________________________|ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |---HostOS---------------------------|----------------------------------------|ÂÂÂÂÂÂÂ |ÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂ _______________________|___________________________________ÂÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ VFIOÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂ KernelÂÂ |_______________________|_____ÂÂÂÂ _________________________|ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ \/ÂÂÂ |ÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂ vfio-pciÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂ |ÂÂÂ vfio_iommu_type1ÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂ |ÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |_______________________|_____|___|_________________________|ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |____________________________________|________________________________________|ÂÂÂÂÂÂÂ |ÂÂÂ |
>>> -----------------------------------------------|-------------------------------------------------|----|--------------
>>>ÂÂÂÂÂÂÂÂÂÂÂÂ ____________________________________\/____ÂÂÂÂ _________________________ÂÂÂÂÂÂÂÂÂÂ ___\/___\/___________
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂ |Â |ÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂ |
>>> Hardware | | | DMA (gIOVA) --|---|-> IOMMU --|---------|->|______| |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂ |ÂÂÂ |_____________________________|ÂÂ |ÂÂÂÂÂ (gIOVA-->hPA)ÂÂÂÂÂ |ÂÂ (hPA) |ÂÂ Physical MemoryÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ | DPUÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂ |_________________________|ÂÂÂÂÂÂÂÂ |_____________________|
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂ |ÂÂÂÂÂÂÂ NVME-ofÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂ |__________________________________|
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |___________________________|______________|
>>> --------------------------------------|------------------------------------------------------------------------------
>>>ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ | TCP (RDMA, and so on)
>>>ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ ______________v__________
>>>ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
>>> Remote storageÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂ Network StorageÂÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |_________________________|
>>>ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ
>>> ---------------------------------------------------------------------------------------------------------------------
>>>
>>>
>>> It is difficult to implement PCIe passthrough live migration.
>>
>> Linux commit 115dcec65f61d53e25e1bed5e380468b30f98b14 ("vfio: Define
>> device migration protocol v2") defines the VFIO migration API and it's
>> implemented by several drivers in the kernel.
>>
>>Can you explain the difficulty of implementing PCIe passthrough live
>> migration in more detail?
>>
>>>
>>>
>>>
>>>ÂÂ
>>> virtio-nvme:
>>> ---------------------------------------------------------------------------------------------------------------------
>>>ÂÂÂÂÂÂÂÂÂÂÂÂ _____________________________________________________________________________
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂ ___________________________________________________________ÂÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ _____________________________________________________Â |ÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ | |ÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂ __________________________________ÂÂÂÂ | |ÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ ______ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ | |ÂÂÂ |ÂÂÂÂÂÂÂ ______
>>> | | | User | | Mem |-----------------------|----|-|----|-----> | |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂ |______|ÂÂÂÂÂÂÂ SPDKÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ | |ÂÂÂ | (gVA) |______|
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ (gVA)ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ | |ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |______|___________________________|ÂÂÂ | |ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |--GuestOS----------|--------------------------------| |ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂ ______v___________________________ÂÂÂÂ | |ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ | VM |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ VFIOÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ | |ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂ UserÂÂÂÂ |ÂÂÂ |ÂÂ KernelÂÂ |___________ÂÂÂÂ __________________|ÂÂÂ | |ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |
>>> | | | | vfio-pci | | | | | | | |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |Â (gIOVA)Â |ÂÂ | vfio_iommu_type1 |ÂÂÂ | |ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |
>>> Software | | | |______|____|___|__________________| | | | | |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |___________________|________________________________| |ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂ ___________________|________________________________Â |ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂ ______v____ÂÂÂÂ __________________ÂÂÂÂ | |ÂÂÂ |ÂÂÂÂÂÂÂ vÂÂÂ v
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |virtio-NVME|ÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ | |ÂÂÂ |ÂÂÂÂÂÂÂ ______
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |Â Instance |ÂÂ |ÂÂÂÂÂ vIOMMUÂÂÂÂÂ |ÂÂÂ | |ÂÂÂ |ÂÂÂÂÂÂ |ÂÂÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂ QEMUÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂ |ÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ --|----|-|----|-----> |______|
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ | vhost-vdpa|ÂÂ |ÂÂ (gIOVA-->gPA)Â |ÂÂÂ | |ÂÂÂ | (gPA)Â |ÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |__(gIOVA)__|ÂÂ |__________________|ÂÂÂ | |ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |__________________|_________________________________| |ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |_______________________|___________________________________|ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |---HostOS---------------------------|----------------------------------------|ÂÂÂÂÂÂÂ |ÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂ _______________________|___________________________________ÂÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ vDPAÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂ KernelÂÂ |_______________________v_____ÂÂÂÂ _________________________|ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂ |ÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂ vdpa-deviceÂÂÂÂÂÂÂÂ |ÂÂÂÂ |ÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ | (Virtual device)ÂÂÂÂÂ |ÂÂÂÂ |ÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |_______________________|_____|___|_________________________|ÂÂÂ |ÂÂÂÂÂÂÂ |ÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |____________________________________|________________________________________|ÂÂÂÂÂÂÂ |ÂÂÂ |
>>> -----------------------------------------------|-------------------------------------------------|----|--------------
>>>ÂÂÂÂÂÂÂÂÂÂÂÂ ____________________________________v_____ÂÂÂÂ _________________________ÂÂÂÂÂÂÂÂÂÂ ___v____v____________
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂ |ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂ |Â |ÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂ |
>>> Hardware | | | DMA (gIOVA) --|---|-> IOMMU --|---------|->|______| |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂ |ÂÂÂ |_____________________________|ÂÂ |ÂÂÂÂÂ (gIOVA-->hPA)ÂÂÂÂÂ |ÂÂ (hPA) |ÂÂ Physical MemoryÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |Â DPUÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂ |_________________________|ÂÂÂÂÂÂÂÂ |_____________________|
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂ |ÂÂÂÂÂÂÂ virtio-nvme-ofÂÂÂÂÂÂÂÂÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂ |__________________________________|
>>>ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂ |___________________________|______________|
>>> --------------------------------------|------------------------------------------------------------------------------
>>>ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ | TCP (RDMA, and so on)
>>>ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ ______________v__________
>>>ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
>>> Remote storageÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂ Network StorageÂÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
>>>ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |_________________________|
>>>ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ
>>> ---------------------------------------------------------------------------------------------------------------------
>>> Based on the vDPA framework, it supports live migration.
>>
>> The two diagrams are quite similar. Did you want to highlight a
>> difference between the two approaches in the diagram?
Â
>I also wondered why virtio-nvme is exactly needed,
Traditional NVME devices can only rely on the VFIO framework for
pcie passthrough in virtualization scenarios.The current VFIO framework
does not support live migration well.
In addition to the VFIO-based PCIe passthrough, virtio-nvme physical
hardware can be combined with the vDPA framework to achieve data
plane acceleration.
>and why one couldn't
>write a different "backend" for an ordinary NVME device, that talks to
>the vdpa-device.
The vDPA (virtio data path acceleration) kernel framework is a pillar in
productizing the end-to-end vDPA solution and it enables NIC vendors
to integrate their vDPA NIC kernel drivers into the framework as part of
their productization efforts.
So NVME can't talks to vDPA framework.
Â
--
Thanks,
Â
David / dhildenb
Â
Â
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]