Re: Re: [virtio-comment] [PROPOSAL] Virtio Over Fabrics(TCP/RDMA)

[zhenwei pi <pizhenwei@bytedance.com>]

On 4/25/23 13:03, Parav Pandit wrote:
[...]
> Example #3 enables to implement virtio backend over fabrics initiator in 
> the user space, which is also a good use case.
Yes.

> It can be also be done in non native virtio backend.
> More below.
>
> > 3, SmartNIC/DPU/vDPA environment. It's possible to convert virtio-pci 
> > request to virtio-of request by hardware, and forward request to 
> > virtio target directly.
> > ÂÂÂÂÂÂÂÂ +----------+ÂÂÂ +----------+ÂÂÂÂÂÂ +----------+
> >  |Map-Reduce| | nginx | ... | processes|
> > ÂÂÂÂÂÂÂÂ +----------+ÂÂÂ +----------+ÂÂÂÂÂÂ +----------+
> > ------------------------------------------------------------
> > HostÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
> > KernelÂÂ +-------+ÂÂÂÂÂÂ +-------+ÂÂÂÂÂÂÂÂÂ +-------+
> > ÂÂÂÂÂÂÂÂÂ | ext4Â |ÂÂÂÂÂÂ | LKCFÂ |ÂÂÂÂÂÂÂÂÂ | HWRNG |
> > ÂÂÂÂÂÂÂÂÂ +-------+ÂÂÂÂÂÂ +-------+ÂÂÂÂÂÂÂÂÂ +-------+
> > ÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
> > ÂÂÂÂÂÂÂÂÂ +-------+ÂÂÂÂÂÂ +-------+ÂÂÂÂÂÂÂÂÂ +-------+
> >  | vdx | |vCrypto| | vRNG |
> > ÂÂÂÂÂÂÂÂÂ +-------+ÂÂÂÂÂÂ +-------+ÂÂÂÂÂÂÂÂÂ +-------+
> > ÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
> > PCI --------------------------------------------------------
> > ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
> > SmartNICÂÂÂÂÂÂÂÂÂÂÂÂ +---------------+
> > ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |virtio HW queue|
> > ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +---------------+
> > ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
> > ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +------+
> > ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |NIC/IB|
> > ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +------+
> > ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +-------------+
> > ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +--------------------->|virtio target|
> > ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +-------------+
> All 3 seems a valid use cases.
>
> Use case 1 and 2 can be achieved directly without involving any 
> mediation layer or any other translation layer (for example virtio to nfs).
> Many blk and file protocols outside of the virtio already exists which 
> achieve this. I don't see virtio being any different to support this in 
> native manner, mainly the blk, fs, crypto device.

I'm working on iSCSI/iSER/NVMe-oF for several years, frankly, I also 
think there is no essential difference between virtio-blk over fabrics 
and NVMe-oF. As far as I know, storage specified protocol is quit common 
and popular, but the other device types seems lack. For example, I don't 
know any protocol to support remote crypto device(please correct me if 
there is already one). I also notice that virtio-camera is reserved, a 
remote camera protocol may be developed to support camera redirection in 
future. virtio-of works on virtio transport layer, it allow us to 
support almost the full virtio device family.

> use case #3 brings additional a benefits at the same time different 
> complexity but sure #3 is also a valid and common use case in our 
> experiences.
>
> In my experience working with FC, iSCSI, FCoE, NVMe RDMA fabrics, iSER,
> A virito fabrics needs a lot of work to reach the scale, resiliency and 
> lastly the security. (step by step...)
>
> My humble suggestion is : pick one transport instead of all at once, 
> rdma being most performant probably the first candidate to see the perf 
> gain for use case #1 and #2 from remote system.
>
> I briefly see your rdma command descriptor example, which is not aligned 
> to 16B. Perf wise it will be poor than nvme rdma fabrics.
Thanks, I'll confirm this with RDMA expert, maybe add a padding field in 
command.

> For PCI transport for net, we intent to start the work to improve 
> descriptors, the transport binding for net device. From our research I 
> see that some abstract virtio descriptors are great today, but if you 
> want to get best out of the system (sw, hw, cpu), such abstraction is 
> not the best. Sharing of "id" all the way to target and bring back is an 
> example of such inefficiency in your example.

Could you please give me more hint about this?
Because requests from virtqueue always have unique id, so the inflight 
virtio-of request use id to distinguish. otherwise, it has no other meaning.

--
zhenwei pi