Re: [PATCH v5 3/7] transport-fabrics: introduce Virtio-oF Protocol Data Unit

[Raphael Norwitz <raphael.norwitz@nutanix.com>]

One suggestion I didnât think of in my prior review.

> On Oct 23, 2023, at 6:46 AM, zhenwei pi <pizhenwei@bytedance.com> wrote:
> 
> Introduce Virtio-oF PDU for Virtio-oF queue, add Stream Data Transfers
> and Keyed Data Transfers mechanism.
> 
> Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
> â
<snip>
> +\begin{lstlisting}
> +struct virtio_of_keyed_desc {
> +        /* the remote address of data */
> +        le64 addr;
> +        /* the length of data */
> +        le32 length;
> +        /* the key to access the remote data */
> +        le32 key;
> +};
> +\end{lstlisting}

Going off my comment here: https://lists.oasis-open.org/archives/virtio-comment/202307/msg00253.html

You could probably save some space space in virtio_of_keyed_desc with a protocol extension where you can pre-register some number of MRs/keys with the target and reference them by index.  For example, you could save 16 bits per descriptor, lowering the total memory footprint by 1/8 by doing something like:

struct virtio_of_small_keyed_desc {
        /* the remote address of data */
        le64 addr;
        /* the length of data */
        le32 length;
        /* the mr_id which maps to a key (and possibly a base address) on the target size */
        le16 mr_hint;
};

If you could bound the size of each MR (to, say 4GB) it could make sense to send a 32 bit offset into the MR and then re-compute the address on the target side.