Re: Re: [virtio-comment] [PROPOSAL] Virtio Over Fabrics(TCP/RDMA)

[zhenwei pi <pizhenwei@bytedance.com>]

On 4/24/23 11:40, Jason Wang wrote:
> On Sun, Apr 23, 2023 at 7:31âPM zhenwei pi <pizhenwei@bytedance.com> wrote:
> > Hi,
> >
> > In the past years, virtio supports lots of device specifications by
> > PCI/MMIO/CCW. These devices work fine in the virtualization environment,
> > and we have a chance to support virtio device family for the
> > container/host scenario.
>
> PCI can work for containers for sure (or does it meet any issue like
> scalability?). It's better to describe what problems you met and why
> you choose this way to solve it.
>
> It's better to compare this with
>
> 1) hiding the fabrics details via DPU
> 2) vDPA
Hi,

Sorry, I missed this part. "Network defined peripheral devices of virtio 
family" is the main purpose of this proposal, this allows us to use many 
types of remote resources which are provided by virtio target.

From the point of my view, there are 3 cases:
1, Host/container scenario. For example, host kernel connects a virtio 
target block service, maps it as a vdx(virtio-blk) device(used by 
Map-Reduce service which needs a fast/large size disk). The host kernel 
also connects a virtio target crypto service, maps it as virtio crypto 
device(used by nginx to accelarate HTTPS). And so on.

        +----------+    +----------+       +----------+
        |Map-Reduce|    |   nginx  |  ...  | processes|
        +----------+    +----------+       +----------+
------------------------------------------------------------
Host         |               |                  |
Kernel   +-------+       +-------+          +-------+
         | ext4  |       | LKCF  |          | HWRNG |
         +-------+       +-------+          +-------+
             |               |                  |
         +-------+       +-------+          +-------+
         |  vdx  |       |vCrypto|          | vRNG  |
         +-------+       +-------+          +-------+
             |               |                  |
             |           +--------+             |
             +---------->|TCP/RDMA|<------------+
                         +--------+
                             |
                         +------+
                         |NIC/IB|
                         +------+
                             |                      +-------------+
                             +--------------------->|virtio target|
                                                    +-------------+

2, Typical virtualization environment. The workloads run in a guest, and 
QEMU handles virtio-pci(or MMIO), and forwards requests to target.
        +----------+    +----------+       +----------+
        |Map-Reduce|    |   nginx  |  ...  | processes|
        +----------+    +----------+       +----------+
------------------------------------------------------------
Guest        |               |                  |
Kernel   +-------+       +-------+          +-------+
         | ext4  |       | LKCF  |          | HWRNG |
         +-------+       +-------+          +-------+
             |               |                  |
         +-------+       +-------+          +-------+
         |  vdx  |       |vCrypto|          | vRNG  |
         +-------+       +-------+          +-------+
             |               |                  |
PCI --------------------------------------------------------
                             |
QEMU                 +--------------+
                     |virtio backend|
                     +--------------+
                             |
                         +------+
                         |NIC/IB|
                         +------+
                             |                      +-------------+
                             +--------------------->|virtio target|
                                                    +-------------+

3, SmartNIC/DPU/vDPA environment. It's possible to convert virtio-pci 
request to virtio-of request by hardware, and forward request to virtio 
target directly.
        +----------+    +----------+       +----------+
        |Map-Reduce|    |   nginx  |  ...  | processes|
        +----------+    +----------+       +----------+
------------------------------------------------------------
Host         |               |                  |
Kernel   +-------+       +-------+          +-------+
         | ext4  |       | LKCF  |          | HWRNG |
         +-------+       +-------+          +-------+
             |               |                  |
         +-------+       +-------+          +-------+
         |  vdx  |       |vCrypto|          | vRNG  |
         +-------+       +-------+          +-------+
             |               |                  |
PCI --------------------------------------------------------
                             |
SmartNIC             +---------------+
                     |virtio HW queue|
                     +---------------+
                             |
                         +------+
                         |NIC/IB|
                         +------+
                             |                      +-------------+
                             +--------------------->|virtio target|
                                                    +-------------+

> > - Theory
> > "Virtio Over Fabrics" aims at "reuse virtio device specifications", and
> > provides network defined peripheral devices.
> > And this protocol also could be used in virtualization environment,
> > typically hypervisor(or vhost-user process) handles request from virtio
> > PCI/MMIO/CCW, remaps request and forwards to target by fabrics.
>
> This requires meditation in the datapath, isn't it?
>
> > - Protocol
> > The detail protocol definition see:
> > https://github.com/pizhenwei/linux/blob/virtio-of-github/include/uapi/linux/virtio_of.h
>
> I'd say a RFC patch for virtio spec is more suitable than the codes.

OK. I'll send a RFC patch for virtio spec later if this proposal is 
acceptable.

[...]

> A quick glance at the code told me it's a mediation layer that convert
> descriptors in the vring to the fabric specific packet. This is the
> vDPA way.
>
> If we agree virtio of fabic is useful, we need invent facilities to
> allow building packet directly without bothering the virtqueue (the
> API is layout independent anyhow).
>
> Thanks

This code describes the case 1[Host/container scenario], also proves 
this case works.
Create a virtqueue in the virtio fabric module, also emulate a 
"virtqueue backend" here, when uplayer kicks vring, the "backend" gets 
notified and builds packet to TCP/RDMA.

[...]

--
zhenwei pi