Re: Re: [virtio-comment] [PROPOSAL] Virtio Over Fabrics(TCP/RDMA)

[Jason Wang <jasowang@redhat.com>]

On Mon, Apr 24, 2023 at 9:38âPM zhenwei pi <pizhenwei@bytedance.com> wrote:
>
>
>
> On 4/24/23 11:40, Jason Wang wrote:
> > On Sun, Apr 23, 2023 at 7:31âPM zhenwei pi <pizhenwei@bytedance.com> wrote:
> >>
> >> Hi,
> >>
> >> In the past years, virtio supports lots of device specifications by
> >> PCI/MMIO/CCW. These devices work fine in the virtualization environment,
> >> and we have a chance to support virtio device family for the
> >> container/host scenario.
> >
> > PCI can work for containers for sure (or does it meet any issue like
> > scalability?). It's better to describe what problems you met and why
> > you choose this way to solve it.
> >
> > It's better to compare this with
> >
> > 1) hiding the fabrics details via DPU
> > 2) vDPA
> >
> Hi,
>
> Sorry, I missed this part. "Network defined peripheral devices of virtio
> family" is the main purpose of this proposal,

This can be achieved by either DPU or vDPA. I think the advantages is,
if we standardize this in the spec, it avoids vendor specific
protocol.

> this allows us to use many
> types of remote resources which are provided by virtio target.
>
>  From the point of my view, there are 3 cases:
> 1, Host/container scenario. For example, host kernel connects a virtio
> target block service, maps it as a vdx(virtio-blk) device(used by
> Map-Reduce service which needs a fast/large size disk). The host kernel
> also connects a virtio target crypto service, maps it as virtio crypto
> device(used by nginx to accelarate HTTPS). And so on.
>
>          +----------+    +----------+       +----------+
>          |Map-Reduce|    |   nginx  |  ...  | processes|
>          +----------+    +----------+       +----------+
> ------------------------------------------------------------
> Host         |               |                  |
> Kernel   +-------+       +-------+          +-------+
>           | ext4  |       | LKCF  |          | HWRNG |
>           +-------+       +-------+          +-------+
>               |               |                  |
>           +-------+       +-------+          +-------+
>           |  vdx  |       |vCrypto|          | vRNG  |
>           +-------+       +-------+          +-------+
>               |               |                  |
>               |           +--------+             |
>               +---------->|TCP/RDMA|<------------+
>                           +--------+
>                               |
>                           +------+
>                           |NIC/IB|
>                           +------+
>                               |                      +-------------+
>                               +--------------------->|virtio target|
>                                                      +-------------+
>
> 2, Typical virtualization environment. The workloads run in a guest, and
> QEMU handles virtio-pci(or MMIO), and forwards requests to target.
>          +----------+    +----------+       +----------+
>          |Map-Reduce|    |   nginx  |  ...  | processes|
>          +----------+    +----------+       +----------+
> ------------------------------------------------------------
> Guest        |               |                  |
> Kernel   +-------+       +-------+          +-------+
>           | ext4  |       | LKCF  |          | HWRNG |
>           +-------+       +-------+          +-------+
>               |               |                  |
>           +-------+       +-------+          +-------+
>           |  vdx  |       |vCrypto|          | vRNG  |
>           +-------+       +-------+          +-------+
>               |               |                  |
> PCI --------------------------------------------------------
>                               |
> QEMU                 +--------------+
>                       |virtio backend|
>                       +--------------+
>                               |
>                           +------+
>                           |NIC/IB|
>                           +------+
>                               |                      +-------------+
>                               +--------------------->|virtio target|
>                                                      +-------------+
>

So it's the job of the Qemu to do the translation from virtqueue to packet here?

> 3, SmartNIC/DPU/vDPA environment. It's possible to convert virtio-pci
> request to virtio-of request by hardware, and forward request to virtio
> target directly.
>          +----------+    +----------+       +----------+
>          |Map-Reduce|    |   nginx  |  ...  | processes|
>          +----------+    +----------+       +----------+
> ------------------------------------------------------------
> Host         |               |                  |
> Kernel   +-------+       +-------+          +-------+
>           | ext4  |       | LKCF  |          | HWRNG |
>           +-------+       +-------+          +-------+
>               |               |                  |
>           +-------+       +-------+          +-------+
>           |  vdx  |       |vCrypto|          | vRNG  |
>           +-------+       +-------+          +-------+
>               |               |                  |
> PCI --------------------------------------------------------
>                               |
> SmartNIC             +---------------+
>                       |virtio HW queue|
>                       +---------------+
>                               |
>                           +------+
>                           |NIC/IB|
>                           +------+
>                               |                      +-------------+
>                               +--------------------->|virtio target|
>                                                      +-------------+
>
> >>
> >> - Theory
> >> "Virtio Over Fabrics" aims at "reuse virtio device specifications", and
> >> provides network defined peripheral devices.
> >> And this protocol also could be used in virtualization environment,
> >> typically hypervisor(or vhost-user process) handles request from virtio
> >> PCI/MMIO/CCW, remaps request and forwards to target by fabrics.
> >
> > This requires meditation in the datapath, isn't it?
> >
> >>
> >> - Protocol
> >> The detail protocol definition see:
> >> https://github.com/pizhenwei/linux/blob/virtio-of-github/include/uapi/linux/virtio_of.h
> >
> > I'd say a RFC patch for virtio spec is more suitable than the codes.
> >
>
> OK. I'll send a RFC patch for virtio spec later if this proposal is
> acceptable.

Well, I think we need to have an RFC first to know if it is acceptable or not.

>
> [...]
>
> >
> > A quick glance at the code told me it's a mediation layer that convert
> > descriptors in the vring to the fabric specific packet. This is the
> > vDPA way.
> >
> > If we agree virtio of fabic is useful, we need invent facilities to
> > allow building packet directly without bothering the virtqueue (the
> > API is layout independent anyhow).
> >
> > Thanks
> >
>
> This code describes the case 1[Host/container scenario], also proves
> this case works.
> Create a virtqueue in the virtio fabric module, also emulate a
> "virtqueue backend" here, when uplayer kicks vring, the "backend" gets
> notified and builds packet to TCP/RDMA.

In this case, it won't perform good. Since it still use virtqueue
which is unnecessary in the datapath for fabric.

Thanks

>
> [...]
>
> --
> zhenwei pi
>