RE: [virtio-comment] Re: [PATCH v2 05/11] transport-fabrics: introduce Keyed Transmission

[Parav Pandit <parav@nvidia.com>]

> From: zhenwei pi <pizhenwei@bytedance.com>
> Sent: Monday, June 5, 2023 4:41 AM
> 
> On 6/5/23 10:41, Parav Pandit wrote:
> >
> >
> > On 5/4/2023 4:19 AM, zhenwei pi wrote:
> >> Keyed transmission is used for message oriented communication(Ex
> >> RDMA), also add virtio-blk read/write 8K example.
> >>
> >> Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
> >> ---
> >
> >> +An example of a virtio-blk write 8K request(message size:
> >> sizeof(Command) +
> >> +4 * sizeof(Descriptor)):
> >> +\begin{lstlisting}
> >> + COMMANDÂÂÂÂÂÂÂÂÂÂÂ +------+
> >> +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |opcode|Â ->Â virtio_of_op_vring
> >> +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +------+
> >> +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |cmd id|Â ->Â 10
> >> +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +------+
> >> +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |length|Â ->Â 0
> >> +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +------+
> >> +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ndesc |Â ->Â 4
> >> +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +------+
> >> + |rsvd |
> >> +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +------+
> >> +
> >> + DESC0ÂÂÂÂÂÂÂÂÂÂÂÂÂ +------+
> >> + |addr | -> 0xffff012345670000
> >> +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +------+
> >> +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |length|Â -> 16 (virtio blk write command)
> >> +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +------+
> >> +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |idÂÂÂ |Â -> 0
> >> +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +------+
> >
> > for RDMA this id is not useful. It can be omitted.
> > still parsing the rest.
> >
> > if we talk blk as an example, above command descriptor can be of 32
> > bytes, such as struct virtio_of_cmd {
> >  ÂÂÂÂu8 opcode;
> >  ÂÂÂÂu8 rsvd;
> >  ÂÂÂÂle16 cmd_id;
> >  ÂÂÂÂu8 inline_desc_cnt;
> >  ÂÂÂÂu8 rsvd[3];
> >  ÂÂÂÂ/* some padding/metadata for long desc list if any */ };
> >
> > struct virtio_of_rdma_desc {
> >  ÂÂÂÂle64 addr;
> >  ÂÂÂÂle32 length;
> >  ÂÂÂÂle32 rdma_key;
> > };
> >
> > struct virtio_rdma_op {
> >  ÂÂÂÂstruct virtio_of_cmd cmd;
> >  ÂÂÂÂstruct virtio_of_rdma_desc desc[1 or 3]; /* count can be
> > negotiated */ };
> >
> > With this a send and receive queue on initiator and target can
> > exchange, cmd descriptor for read/writes.
> >
> 
> Hi,
> 
> Do you mean that separating a Virtio Over RDMA queue into 2 QP, one for
> sending, another one for receiving?
> 
No. just one QP.

Initiator_QP_A -> target_QP_B.

When initiator QP A sends 32B cmd, it lands in the target QP B's receive queue.

After this target can do one or more read/write DMA using RDMA read/write from the initiator's memory.

Finally target_QP_B sends 8B completion, it arrives in the QP_A's receive queue.