Re: [virtio-comment] Re: [PATCH v2 05/11] transport-fabrics: introduce Keyed Transmission

[zhenwei pi <pizhenwei@bytedance.com>]

On 6/5/23 10:41, Parav Pandit wrote:
> On 5/4/2023 4:19 AM, zhenwei pi wrote:
> > Keyed transmission is used for message oriented communication(Ex RDMA),
> > also add virtio-blk read/write 8K example.
> >
> > Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
> > ---
>
> > +An example of a virtio-blk write 8K request(message size: 
> > sizeof(Command) +
> > +4 * sizeof(Descriptor)):
> > +\begin{lstlisting}
> > + COMMANDÂÂÂÂÂÂÂÂÂÂÂ +------+
> > +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |opcode|Â ->Â virtio_of_op_vring
> > +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +------+
> > +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |cmd id|Â ->Â 10
> > +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +------+
> > +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |length|Â ->Â 0
> > +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +------+
> > +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ndesc |Â ->Â 4
> > +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +------+
> > + |rsvd |
> > +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +------+
> > +
> > + DESC0ÂÂÂÂÂÂÂÂÂÂÂÂÂ +------+
> > + |addr | -> 0xffff012345670000
> > +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +------+
> > +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |length|Â -> 16 (virtio blk write command)
> > +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +------+
> > +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |idÂÂÂ |Â -> 0
> > +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +------+
>
> for RDMA this id is not useful. It can be omitted.
> still parsing the rest.
>
> if we talk blk as an example, above command descriptor can be of 32 bytes,
> such as
> struct virtio_of_cmd {
>  ÂÂÂÂu8 opcode;
>  ÂÂÂÂu8 rsvd;
>  ÂÂÂÂle16 cmd_id;
>  ÂÂÂÂu8 inline_desc_cnt;
>  ÂÂÂÂu8 rsvd[3];
>  ÂÂÂÂ/* some padding/metadata for long desc list if any */
> };
>
> struct virtio_of_rdma_desc {
>  ÂÂÂÂle64 addr;
>  ÂÂÂÂle32 length;
>  ÂÂÂÂle32 rdma_key;
> };
>
> struct virtio_rdma_op {
>  ÂÂÂÂstruct virtio_of_cmd cmd;
>  ÂÂÂÂstruct virtio_of_rdma_desc desc[1 or 3]; /* count can be negotiated */
> };
>
> With this a send and receive queue on initiator and target can exchange, 
> cmd descriptor for read/writes.

Hi,

Do you mean that separating a Virtio Over RDMA queue into 2 QP, one for 
sending, another one for receiving?

> RDMA allows mapping memory and also chaining it with next send.
> This way, memory from 1B to 4GB can be represented using single rdma key 
> for data DMA (read or write).
>
> Completion is similarly 8B with status + cmd_id of constant size can be 
> received in an RQ.
>
> This is 1 RTT from initiator to target for cmd and response for whole 
> 4GB data transfer.
> Depending on the data size, memory pressure, sharing, outstanding 
> commands, etc target can read/write data from initiator memory using 
> RDMA read/write addresses.
>
> With this target can also implement poll or event.
>
> RDMA writes do not guarantee data placement visibility in same order on 
> the responder side as what is send on the requester side.
>
> I ran out of time. Will review more later in the week.
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: 
> https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/

--
zhenwei pi