Re: Re: [virtio-comment] About adding a new device type virtio-nvme

[äèä <houyingle@sudoinfotech.com>]

Wed, 18 Jan 2023 12:09:59 +0200, Max Gurtovoy wrote:
>On 18/01/2023 5:23, äèä wrote:

>> On Tue, 17 Jan 2023 19:19:59 +0200, Max Gurtovoy wrote:

>>

>>> On 17/01/2023 4:04, äèä wrote:

>>>> On Wed, 11 Jan 2023 10:16:55 -0500, Stefan wrote:

>>>>>> On Wed, Jan 11, 2023 at 11:21:35AM +0800, äèä wrote:

>>>>>> As we know, nvme has more features than virtio-blk. For example, with the development of virtualization IO offloading to hardware, virtio-blk and NVME-OF offloading to hardware >are developing rapidly. So if virtio and nvme are combined into Virtio-NvMe, Is it necessary to add a device type Virtio-NvMe ?

>>>>ÂÂÂ

>>>>

>>>>> Hi,

>>>>> In theory, yes, virtio-nvme can be done. The question is why do it?

>>>>> NVMe already provides a PCI hardware spec for software and hardware

>>>>> implementations to follow. An NVMe PCI device can be exposed to the

>>>>> guest and modern operating systems recognize it without requiring new

>>>>> drivers.

>>>>> The value of VIRTIO here is probably in the deep integration into the

>>>>> virtualization stack with vDPA, vhost, etc. A virtio-nvme device can use

>>>>> all these things whereas a PCI device needs to do everything from

>>>>> scratch.

>>>> The NVME technology and ecosystem are complete. However, in virtualization scenarios, NVME devices can only use PCIe pass-through . When NVME and virtio combine to connect to the vDPA ecosystem, live migration is supported.

>>>>> Let's not forget that virtio-blk is widely used and new commands are

>>>>> being added as needed. Which NVMe features are you missing in

>>>>> virtio-blk?

>>>> With the introduction of the concept of DPU, a large number of vendors are offloading virtual devices to hardware. The back-end of Virtio-blk does not support remote storage. Therefore, Virtio-Nvme-of can well combine the advantages of remote storage and virtio live migration

>>>>> I guess this is why virtio-nvme hasn't been done before: people who want

>>>> NVMe can already do NVMe PCI, people who want VIRTIO can use virtio-blk,

>>>>> and so there hasn't been a great need to combine VIRTIO and NVMe yet.

>>>>> What advantages do you see in having virtio-nvme?

>>>> virtio-nvmeÂadvantages :

>>>> 1)Â live migration

>>>ÂÂ

>>> This is WIP and will use VFIO live migration framework.

>> Yes, VFIO live migration framework is WIP, but I still think vdpa is a friendlier framework.



>Not sure what you consider friendly ?

My personal opinion: VFIO live migration requires device design requirements.
But vDPA-based live migration, the software-abstracted vDPA device in the vDPA
framework can do some state recording, The design requirements for virtio devices
that are offloading to hardware may be lower.

Â


>The community agreed that in SR-IOV - VF migration is done via PF interface.



>Any device specific migration (e.g. vdpa/virtio)Â is not as generic as

>VFIO migration. Also it will be maintained by a smaller group of engineers.


>If you would like to use vdpa - I suggest using virtio-blk and not
>inventing virtio-nvme device that will for sure be with less feature set
>than pure NVMe.



>In case you're missing some feature in virtio-blk that exist in NVMe,
>you're welcome to submit a proposal to the technical group with that
>feature.


Yes, this is good advice.
virtio-blk adds Fabrics related commands to enable virtio-blk to support 
virtio-blk-of (over Fabric), I wonder if it is feasible.




>NVIDIA also has a DPU based physical Virtio-blk device (NVIDIA'S
>virtio-blk SNAP) that support SR-IOV and remote storage access.


For remote storage access, how is the physical Virtio-blk device's back-end implemented?
What protocol is used? 
Is it an open source solution?




>Live migration specification is WIP in both NVMe and Virtio working
>groups. I can't say who will be merge first.

Yes, I agree with you on that point.



>>>ÂÂ
>>>> 2)Â support remote storage

>>> There are solutions today that can use remote storage as an NVMe

>>> Namespace. For example, DPU based NVMe device such as NVIDIA'S NVMe SNAP

>>> device.

>>

>>

>> Yes, you're right. Nvme has a built-in advantage over virtio-blk hardware offloading.

>> The reason why I propose Virtio-NVMe is to combine nvme and virtio, so that NVME

>> can adapt to virtio ecosystem based on virtio interface specifications, such as vdpa.