Re: RE: [virtio-comment] Re: [PATCH v2 05/11] transport-fabrics: introduce Keyed Transmission

[zhenwei pi <pizhenwei@bytedance.com>]

On 6/5/23 19:45, Parav Pandit wrote:
> > From: zhenwei pi <pizhenwei@bytedance.com>
> > Sent: Monday, June 5, 2023 4:41 AM
> >
> > On 6/5/23 10:41, Parav Pandit wrote:
> > > On 5/4/2023 4:19 AM, zhenwei pi wrote:
> > > > Keyed transmission is used for message oriented communication(Ex
> > > > RDMA), also add virtio-blk read/write 8K example.
> > > >
> > > > Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
> > > > ---
> > >
> > > > +An example of a virtio-blk write 8K request(message size:
> > > > sizeof(Command) +
> > > > +4 * sizeof(Descriptor)):
> > > > +\begin{lstlisting}
> > > > + COMMANDÂÂÂÂÂÂÂÂÂÂÂ +------+
> > > > +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |opcode|Â ->Â virtio_of_op_vring
> > > > +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +------+
> > > > +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |cmd id|Â ->Â 10
> > > > +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +------+
> > > > +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |length|Â ->Â 0
> > > > +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +------+
> > > > +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ndesc |Â ->Â 4
> > > > +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +------+
> > > > + |rsvd |
> > > > +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +------+
> > > > +
> > > > + DESC0ÂÂÂÂÂÂÂÂÂÂÂÂÂ +------+
> > > > + |addr | -> 0xffff012345670000
> > > > +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +------+
> > > > +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |length|Â -> 16 (virtio blk write command)
> > > > +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +------+
> > > > +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |idÂÂÂ |Â -> 0
> > > > +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +------+
> > >
> > > for RDMA this id is not useful. It can be omitted.
> > > still parsing the rest.
> > >
> > > if we talk blk as an example, above command descriptor can be of 32
> > > bytes, such as struct virtio_of_cmd {
> > >   ÂÂÂÂu8 opcode;
> > >   ÂÂÂÂu8 rsvd;
> > >   ÂÂÂÂle16 cmd_id;
> > >   ÂÂÂÂu8 inline_desc_cnt;
> > >   ÂÂÂÂu8 rsvd[3];
> > >   ÂÂÂÂ/* some padding/metadata for long desc list if any */ };
> > >
> > > struct virtio_of_rdma_desc {
> > >   ÂÂÂÂle64 addr;
> > >   ÂÂÂÂle32 length;
> > >   ÂÂÂÂle32 rdma_key;
> > > };
> > >
> > > struct virtio_rdma_op {
> > >   ÂÂÂÂstruct virtio_of_cmd cmd;
> > >   ÂÂÂÂstruct virtio_of_rdma_desc desc[1 or 3]; /* count can be
> > > negotiated */ };
> > >
> > > With this a send and receive queue on initiator and target can
> > > exchange, cmd descriptor for read/writes.
> >
> > Hi,
> >
> > Do you mean that separating a Virtio Over RDMA queue into 2 QP, one for
> > sending, another one for receiving?
> No. just one QP.
>
> Initiator_QP_A -> target_QP_B.
>
> When initiator QP A sends 32B cmd, it lands in the target QP B's receive queue.
>
> After this target can do one or more read/write DMA using RDMA read/write from the initiator's memory.

Hi, I have several questions:
1, how to tell the target to read/write DMA using RDMA read/write? is 
virtio_of_rdma_desc missing?

2, if several virtio_of_rdma_desc arrives, the target need to 
distinguish READ * m + WRITE * n descriptors. but *flags* field has been 
removed ...

3, if I understand correctly, Initiator_QP_A -> Target_QP_B(CMD), 
Target_QP_B(RDMA READ), Target_QP_B(RDMA WRITE), Target_QP_B -> 
Initiator_QP_A(COMP). this uses 4 RTT.

> Finally target_QP_B sends 8B completion, it arrives in the QP_A's receive queue.

--
zhenwei pi