Re: [virtio-comment] [PATCH v4 7/7] transport-fabrics: introduce keyed eager buffers

[Raphael Norwitz <raphael.norwitz@nutanix.com>]

I do think this is a very valuable addition to the protocol but it introduces complication.

The fundamental drawback is that itâs really expensive to allocate and pre-post enough memory in the target. In theory, a guest could submit <VQ size> <max IO size> IOs on all VQ connections. Assuming a 128 VQ size and 1MB IOs, thatâs 128 MB per connection. That may be great for some use cases but doesnât scale.

With the proposed design Iâd worry about memory fragmentation. Maybe it would be better to have eager buffer regions for different IO size buckets per VQ? While slightly less memory efficient it would be much easier to manage.

Again, if this is just for storage device write path, why not try extend iSER or NVMF with this functionality?

> On Jun 26, 2023, at 3:25 AM, zhenwei pi <pizhenwei@bytedance.com> wrote:
> 
> The Virtio-oF device instance can expose keyed eager buffers, the Virtio-oF
> initiator can write device-readable virtqueue buffers and send VQ command
> in a single RTT on the fabric, this makes better performance.
> 
> Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
> ---
> transport-fabrics.tex | 81 +++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 81 insertions(+)
> 
> diff --git a/transport-fabrics.tex b/transport-fabrics.tex
> index f7f29cf..b0ccfb1 100644
> --- a/transport-fabrics.tex
> +++ b/transport-fabrics.tex
> @@ -194,6 +194,8 @@ \subsubsection{Opcodes}\label{sec:Virtio Transport Options / Virtio Over Fabrics
> #define virtio_of_op_get_feature           0x0004
> #define virtio_of_op_set_feature           0x0005
> #define virtio_of_op_get_keyed_num_descs   0x0100
> +#define virtio_of_op_get_keyed_eager_buf   0x0102
> +#define virtio_of_op_vq_keyed_eager_buf    0x0ffe
> #define virtio_of_op_vq                    0x0fff
> #define virtio_of_op_get_vendor_id         0x1000
> #define virtio_of_op_get_device_id         0x1001
> @@ -349,6 +351,9 @@ \subsubsection{Opcodes}\label{sec:Virtio Transport Options / Virtio Over Fabrics
> \begin{lstlisting}
> /* support virtio_of_op_get_keyed_num_descs to get the maximum number of keyed descriptors */
> #define VIRTIO_OF_F_KEYED_NUM_DESCS            0
> +
> +/* support virtio_of_op_get_keyed_eager_buf to get the eager buffers of the Virtio-oF virtqueues */
> +#define VIRTIO_OF_F_KEYED_EAGER_BUF            1
> \end{lstlisting}
> 
> \paragraph{Set Feature Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Set Feature Command}
> @@ -407,6 +412,82 @@ \subsubsection{Opcodes}\label{sec:Virtio Transport Options / Virtio Over Fabrics
> };
> \end{lstlisting}
> 
> +\paragraph{Get Keyed Eager Buffers Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Get Keyed Eager Buffers Command}
> +A Virtio-oF virtqueue may expose 0 or more keyed buffers that allow the Virtio-oF initiator to write directly to the Virtio-oF device instance.
> +The Keyed Eager Buffers mechanism reduces round-trip time on the fabrics.
> +The Get Keyed Eager Buffers Command is used to get the eager buffers of a Virtio-oF virtqueue,
> +and it is executed on the Virtio-oF control queue.
> +
> +The structure of the Get Keyed Deager Buffers Command structure is as follows:

s/Deager/Eager

> +\begin{lstlisting}
> +struct virtio_of_command_get_keyed_eager_buf {
> +        /* opcode is virtio_of_op_get_keyed_eager_buf */
> +        le16 opcode;
> +        /* unique ID for all in-flight commands */
> +        le16 command_id;
> +        /* the Virtio-oF virtqueue index */
> +        le16 vq_index;
> +        /* the maximum number of eager buffer descriptors to obtain */
> +        u8 descs;
> +        u8 reserved[9];
> +};
> +\end{lstlisting}
> +
> +The Get Keyed Eager Buffers Body is a contiguous memory of \texttt{descs} elements (virtio_of_keyed_desc type).
> +
> +The structure of the Get Keyed Eager Buffers Completion is as follows:
> +\begin{lstlisting}

Assuming you are going to have the initiator RDMA write to this memory, shouldnât this include the MR key?

> +struct virtio_of_completion_get_keyed_eager_buf {
> +        le16 status;
> +        le16 command_id;
> +        /* the number of eager buffer descriptors */
> +        u8 descs;
> +        u8 reserved[11];
> +};
> +\end{lstlisting}
> +
> +Note: multiple keyed eager buffers compose a logical memory range sequentially.
> +To locate a position within the range, use the offset and length values as follows:
> +\begin{lstlisting}
> +    +-----------+-----------+     +-----------+
> +    |addr0, len0|addr1, len1| ... |addrN, lenN|
> +    +-----------+-----------+     +-----------+
> +                   |                    |
> +                   \---offset, length---/
> +\end{lstlisting}
> +
> +\paragraph{VQ Keyed Eager Buffers Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / VQ Keyed Eager Buffers Command}
> +The Virtio-oF initiator must write device-readable virtqueue buffers into the Virtio-oF virtqueue eager buffers,
> +then issue The VQ Keyed Eager Buffers Command to transmit virtqueue buffers.
> +This command is executed on the Virtio-oF virtqueue only.
> +
> +The structure of the VQ Keyed Eager Buffers Command structure is as follows:
> +\begin{lstlisting}
> +struct virtio_of_command_vq_keyed_eager_buf {
> +        /* opcode is virtio_of_op_vq_keyed_eager_buf */
> +        le16 opcode;
> +        /* unique ID for all in-flight commands */
> +        le16 command_id;
> +        /* the offset of Virtio-oF virtqueue eager buffers */
> +        le32 out_offset;
> +        /* the length of device-readable virtqueue buffers */
> +        le32 out_length;
> +        /* the length of device-writable virtqueue buffers */
> +        le32 in_length;
> +};
> +\end{lstlisting}
> +
> +The structure of the VQ Keyed Eager Buffers Completion is as follows:
> +\begin{lstlisting}
> +struct virtio_of_completion_vq_keyed_eager_buf {
> +        le16 status;
> +        le16 command_id;
> +        /* the length of virtqueue buffers from Virtio-oF target */
> +        le32 length;
> +        u8 reserved[8];
> +};
> +\end{lstlisting}
> +
> \paragraph{VQ Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / VQ Command}
> The VQ Command is used to transmit virtqueue buffers for Virtio-oF virtqueue only.
> 
> -- 
> 2.25.1
> 
> 
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
> 
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
> 
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
> 
>