Re: Re: [PATCH v5 3/7] transport-fabrics: introduce Virtio-oF Protocol Data Unit

[zhenwei pi <pizhenwei@bytedance.com>]

On 12/12/23 14:22, Raphael Norwitz wrote:
> One suggestion I didnât think of in my prior review.
>
> > On Oct 23, 2023, at 6:46 AM, zhenwei pi <pizhenwei@bytedance.com> wrote:
> >
> > Introduce Virtio-oF PDU for Virtio-oF queue, add Stream Data Transfers
> > and Keyed Data Transfers mechanism.
> >
> > Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
> > â
> <snip>
> > +\begin{lstlisting}
> > +struct virtio_of_keyed_desc {
> > +        /* the remote address of data */
> > +        le64 addr;
> > +        /* the length of data */
> > +        le32 length;
> > +        /* the key to access the remote data */
> > +        le32 key;
> > +};
> > +\end{lstlisting}
>
> Going off my comment here: https://lists.oasis-open.org/archives/virtio-comment/202307/msg00253.html
>
> You could probably save some space space in virtio_of_keyed_desc with a protocol extension where you can pre-register some number of MRs/keys with the target and reference them by index.  For example, you could save 16 bits per descriptor, lowering the total memory footprint by 1/8 by doing something like:
>
> struct virtio_of_small_keyed_desc {
>          /* the remote address of data */
>          le64 addr;
>          /* the length of data */
>          le32 length;
>          /* the mr_id which maps to a key (and possibly a base address) on the target size */
>          le16 mr_hint;
> };
>
> If you could bound the size of each MR (to, say 4GB) it could make sense to send a 32 bit offset into the MR and then re-compute the address on the target side.

In my imagination, for example, the target reports the max Keyed desc as 4:
1, the initiator side has less segments(Ex, 3), the initiator just sends 
3 Keyed desc.
2, the initiator side has more segments(Ex, 6), the initiator side has 
to allocate a large memory, copy the suitable segments to the allocated 
memory(or copy from).
Ex, OUT[0] 16B, OUT[1] 4K, OUT[2] 4K, OUT[3] 16K, OUT[4] 8K, IN[5] 1B
then the initiator side needs allocate 8193 memory, copies OUT[0], 
OUT[1] and OUT[2] into new memory, finally sends 4 Keyed desc.

From my knowledge, most virtio devices usually use fixed segments(not a 
hard limitation), virtio-blk also reports max_seg in config. So I 
suppose the target side could report reasonable value to avoid memory copy.

--
zhenwei pi