[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]
Subject: Re: [RFC v2] virtio-net: Add RoCE (RDMA over Converged Ethernet) support
On Thu, Aug 4, 2022 at 4:30 PM Jason Wang <jasowang@redhat.com> wrote:
>
> On Wed, May 11, 2022 at 5:59 PM Xie Yongji <xieyongji@bytedance.com> wrote:
> >
> > Hi all,
> >
>
> Not very familiar with ROCE, try to give some comments from general
> virtio level.
>
Thank you!
> > This RFC aims to introduce our recent work on enabling RoCE support
> > for virtio-net device.
>
> We need to clarify the version of ROCE, is it ROCEv2 or not?
>
Yes, it's RoCE v2.
> >
> > To support RoCE, three types of virtqueues including RDMA send virtqueue,
> > RDMA receive virtqueue and RDMA completion virtqueue are introduced.
> > And control virtqueue is reused to support the RDMA control messages.
> >
> > Now we support some basic RDMA semantics such as send/receive
> > and read/write operation.
>
> It would be better to explain the advantages of this over the existing
> pvrdma approach. I guess one advantage is that using virtio makes it
> easier to connect to a userspace dataplane through vDPA/vhost-user?
>
Yes, this is one advantage. Another one is that we don't need a
physical RDMA-capable NIC.
> >
> > To test with our demo:
> >
> > 1. Build Guest kernel [1] with config INFINIBAND_VIRTIO_RDMA
> >
> > 2. Build QEMU [2] with config VHOST_USER_RDMA
> >
> > 3. Build rdma-core [3]
> >
> > 4. Build and install DPDK (NOTE that we only tested on DPDK 20.11.3)
> >
> > 5. Build vhost-user-rdma [4]
> >
> > 6. Run vhost-user-rdma with command:
> > $ ./vhost-user-rdma --vdev 'net_tap0' --lcore '1-3' -- -s '/tmp/vhost-rdma0'
> >
> > 7. Run qemu with command:
> > $ qemu-system-x86_64 -chardev socket,path=/tmp/vhost-rdma0,id=vrdma \
> > -device vhost-user-rdma-pci,page-per-vq,chardev=vrdma ...
>
> It would be better to give some performance numbers (or even compare
> it with pvrdma).
>
OK, will do it in v3.
> >
> > [1] https://github.com/bytedance/linux/tree/virtio-net-roce
> > [2] https://github.com/bytedance/qemu/tree/vhost-user-rdma
> > [3] https://github.com/YongjiXie/rdma-core/tree/virtio-rdma
> > [4] https://github.com/YongjiXie/vhost-user-rdma
> >
> > We have already tested it with ibv_rc_pingpong, ibv_ud_pingpong and some
> > others in rdma-core.
> >
> > TODO:
> >
>
> And we'd better consider the live migration support. Having a quick
> glance, it looks to me trapping the cvq is sufficient?
>
I'm not sure. Each QP has its own state machine, which may also
require save & restore.
> > 1. Add support for Base Memory Management Extensions
> >
> > 2. Add support for atomic operation
> >
> > 3. Add support for SRQ
> >
> > 4. Add support for virtqueue resize
>
> Note that this is already supported by the spec via virtqueue reset.
>
OK.
> >
> > 5. Add support for enabling/disabling virtqueue at runtime
>
> I guess virtqueue reset could help in this case.
>
We might need to do some extension since we want to free the resources
when disabling the queue.
> >
> > Please review, thanks!
> >
> > V1 to V2:
> > - Rework the implementation via extending virtio-net instead of
> > introducing a new device type [Jason]
> > - Add address handle support
> >
> > Signed-off-by: Xie Yongji <xieyongji@bytedance.com>
> > Co-developed-by: Wei Junji <weijunji@bytedance.com>
> > Signed-off-by: Wei Junji <weijunji@bytedance.com>
> > ---
> > content.tex | 858 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
> > 1 file changed, 854 insertions(+), 4 deletions(-)
>
> I wonder if there's some open-source ROCE transport device API that we
> can re-use then we can just behave like a transport layer instead of
> inventing new commands.
>
That's would be better. But I didn't find one.
> >
> > diff --git a/content.tex b/content.tex
> > index 7508dd1..646d82a 100644
> > --- a/content.tex
> > +++ b/content.tex
> > @@ -3008,7 +3008,10 @@ \section{Network Device}\label{sec:Device Types / Network Device}
> > placed in one virtqueue for receiving packets, and outgoing
> > packets are enqueued into another for transmission in that order.
> > A third command queue is used to control advanced filtering
> > -features.
> > +features. And if RoCE (RDMA over Converged Ethernet) capability
> > +is enabled, the virtio network device can also support transmitting
> > +and receiving RDMA message through RDMA send virtqueue, RDMA receive
> > +virtqueue and RDMA completion virtqueue.
> >
> > \subsection{Device ID}\label{sec:Device Types / Network Device / Device ID}
> >
> > @@ -3023,13 +3026,24 @@ \subsection{Virtqueues}\label{sec:Device Types / Network Device / Virtqueues}
> > \item[2(N-1)] receiveqN
> > \item[2(N-1)+1] transmitqN
> > \item[2N] controlq
> > +\item[2N+1] rdma_completeq1
> > +\item[\ldots]
> > +\item[2N+M] rdma_completeqM
> > +\item[2N+M+1] rdma_transmitq1
> > +\item[2N+M+2] rdma_receiveq1
> > +\item[\ldots]
> > +\item[2N+M+2L-1] rdma_transmitqL
> > +\item[2N+M+2L] rdma_receiveqL
> > \end{description}
> >
> > N=1 if neither VIRTIO_NET_F_MQ nor VIRTIO_NET_F_RSS are negotiated, otherwise N is set by
> > - \field{max_virtqueue_pairs}.
> > + \field{max_virtqueue_pairs}. M is set by \field{max_rdma_cqs} and L is set by
> > + \field{max_rdma_qps}.
> >
> > controlq only exists if VIRTIO_NET_F_CTRL_VQ set.
> >
> > + rdma_completeq, rdma_transmitq and rdma_receiveq only exist if VIRTIO_NET_F_ROCE set
> > +
> > \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits}
> >
> > \begin{description}
> > @@ -3084,6 +3098,9 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits
> > \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
> > channel.
> >
> > +\item[VIRTIO_NET_F_ROCE(55)] Device supports RoCE (RDMA over Converged Ethernet)
> > + capability.
> > +
> > \item[VIRTIO_NET_F_HOST_USO (56)] Device can receive USO packets. Unlike UFO
> > (fragmenting the packet) the USO splits large UDP packet
> > to several segments when each of these smaller packets has UDP header.
> > @@ -3129,6 +3146,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device
> > \item[VIRTIO_NET_F_GUEST_ANNOUNCE] Requires VIRTIO_NET_F_CTRL_VQ.
> > \item[VIRTIO_NET_F_MQ] Requires VIRTIO_NET_F_CTRL_VQ.
> > \item[VIRTIO_NET_F_CTRL_MAC_ADDR] Requires VIRTIO_NET_F_CTRL_VQ.
> > +\item[VIRTIO_NET_F_ROCE] Requires VIRTIO_NET_F_CTRL_VQ.
> > \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
> > \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
> > \end{description}
> > @@ -3190,6 +3208,8 @@ \subsection{Device configuration layout}\label{sec:Device Types / Network Device
> > u8 rss_max_key_size;
> > le16 rss_max_indirection_table_length;
> > le32 supported_hash_types;
> > + le32 max_rdma_qps;
> > + le32 max_rdma_cps;
> > };
> > \end{lstlisting}
> > The following field, \field{rss_max_key_size} only exists if VIRTIO_NET_F_RSS or VIRTIO_NET_F_HASH_REPORT is set.
> > @@ -3204,11 +3224,23 @@ \subsection{Device configuration layout}\label{sec:Device Types / Network Device
> > Field \field{supported_hash_types} contains the bitmask of supported hash types.
> > See \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled hash types} for details of supported hash types.
> >
> > +Field \field{max_rdma_qps} only exists if VIRTIO_NET_F_ROCE is set.
> > +It specifies the maximum number of queue pairs (send virtqueue and receive virtqueue) for RoCE usage.
> > +
> > +Field \field{max_rdma_cqs} only exists if VIRTIO_NET_F_ROCE is set.
> > +It specifies the maximum number of completion virtqueue for RoCE usage.
> > +
> > \devicenormative{\subsubsection}{Device configuration layout}{Device Types / Network Device / Device configuration layout}
> >
> > The device MUST set \field{max_virtqueue_pairs} to between 1 and 0x8000 inclusive,
> > if it offers VIRTIO_NET_F_MQ.
> >
> > +The device MUST set \field{max_rdma_qps} to between 1 an 16384 inclusive,
> > +if it offers VIRTIO_NET_F_ROCE.
>
> I wonder why 16384 is chosen here?
>
Since the max queue number is limited to 65536 and we have three types
of queue, the queue number should be less than 65536 / 3. We choose
65536 / 4 here.
> > +
> > +The device MUST set \field{max_rdma_cqs} to between 1 an 16384 inclusive,
> > +if it offers VIRTIO_NET_F_ROCE.
> > +
> > The device MUST set \field{mtu} to between 68 and 65535 inclusive,
> > if it offers VIRTIO_NET_F_MTU.
> >
> > @@ -3306,6 +3338,12 @@ \subsection{Device Initialization}\label{sec:Device Types / Network Device / Dev
> > \item If the VIRTIO_NET_F_CTRL_VQ feature bit is negotiated,
> > identify the control virtqueue.
> >
> > +\item If the VIRTIO_NET_F_ROCE feature bit is negotiated,
> > + identify the the RDMA completion virtqueues, up to max_rdma_cqs.
> > +
> > +\item If the VIRTIO_NET_F_ROCE feature bit is negotiated,
> > + identify the the RDMA send and receive virtqueues, up to max_rdma_qps.
> > +
> > \item Fill the receive queues with buffers: see \ref{sec:Device Types / Network Device / Device Operation / Setting Up Receive Buffers}.
> >
> > \item Even with VIRTIO_NET_F_MQ, only receiveq1, transmitq1 and
> > @@ -4007,6 +4045,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> > u8 command;
> > u8 command-specific-data[];
> > u8 ack;
> > + u8 ack-specific-data[];
> > };
> >
> > /* ack values */
> > @@ -4015,8 +4054,8 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> > \end{lstlisting}
> >
> > The \field{class}, \field{command} and command-specific-data are set by the
> > -driver, and the device sets the \field{ack} byte. There is little it can
> > -do except issue a diagnostic if \field{ack} is not
> > +driver, and the device sets the \field{ack} byte and ack-specific-data. There
> > +is little it can do except issue a diagnostic if \field{ack} is not
> > VIRTIO_NET_OK.
> >
> > \paragraph{Packet Receive Filtering}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Packet Receive Filtering}
> > @@ -4463,6 +4502,534 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> > according to the native endian of the guest rather than
> > (necessarily when not using the legacy interface) little-endian.
> >
> > +\paragraph{RoCE Configuration}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / RoCE Configuration}
> > +
> > +If the driver negotiates the VIRTIO_NET_F_ROCE feature bit (depends on VIRTIO_NET_F_CTRL_VQ),
> > +it can send control commands for RoCE usage. The following commands are defined now:
> > +
> > +\begin{lstlisting}
> > +#define VIRTIO_NET_CTRL_ROCE 6
> > + #define VIRTIO_NET_CTRL_ROCE_QUERY_DEVICE 0
> > + #define VIRTIO_NET_CTRL_ROCE_QUERY_PORT 1
> > + #define VIRTIO_NET_CTRL_ROCE_CREATE_CQ 2
> > + #define VIRTIO_NET_CTRL_ROCE_DESTROY_CQ 3
> > + #define VIRTIO_NET_CTRL_ROCE_CREATE_PD 4
> > + #define VIRTIO_NET_CTRL_ROCE_DESTROY_PD 5
> > + #define VIRTIO_NET_CTRL_ROCE_GET_DMA_MR 6
> > + #define VIRTIO_NET_CTRL_ROCE_REG_USER_MR 7
> > + #define VIRTIO_NET_CTRL_ROCE_DEREG_MR 8
> > + #define VIRTIO_NET_CTRL_ROCE_CREATE_QP 9
> > + #define VIRTIO_NET_CTRL_ROCE_MODIFY_QP 10
> > + #define VIRTIO_NET_CTRL_ROCE_QUERY_QP 11
> > + #define VIRTIO_NET_CTRL_ROCE_DESTROY_QP 12
> > + #define VIRTIO_NET_CTRL_ROCE_CREATE_AH 13
> > + #define VIRTIO_NET_CTRL_ROCE_DESTROY_AH 14
> > + #define VIRTIO_NET_CTRL_ROCE_ADD_GID 15
> > + #define VIRTIO_NET_CTRL_ROCE_DEL_GID 16
> > + #define VIRTIO_NET_CTRL_ROCE_REQ_NOTIFY_CQ 17
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[VIRTIO_NET_CTRL_ROCE_QUERY_DEVICE] Query the attributes of device.
> > + No command-specific-data;
> > + the ack-specific-data is \field{struct virtio_rdma_ack_query_device}.
> > +
> > +\begin{lstlisting}
> > +struct virtio_rdma_ack_query_device {
> > +#define VIRTIO_IB_DEVICE_RC_RNR_NAK_GEN (1 << 0)
>
> What's the meaning of this capability?
>
It indicates whether the device supports RNR-NAK generation for RC
QPs. I will add some comments.
> > + /* Capabilities mask */
> > + le64 device_cap_flags;
>
> Will this introduce a migration compatibility issue? E.g src and dst
> have the same features but different capabilities.
>
Should this be controlled by hypervisor since all capabilities is
emulated by software.
> > + /* Largest contiguous block that can be registered */
> > + le64 max_mr_size;
> > + /* Supported memory shift sizes */
> > + le64 page_size_cap;
> > + /* Hardware version */
> > + le32 hw_ver;
>
> What did "hardware version" mean? Is this something that is defined in
> the IB spec?
>
Yes, it's defined in IB spec.
> > + /* Maximum number of outstanding Work Requests (WR) on Send Queue (SQ) and Receive Queue (RQ) */
> > + le32 max_qp_wr;
>
> Is this implied in the virtqueue size? If not, why?
>
Yes. Will remove it.
> > + /* Maximum number of scatter/gather (s/g) elements per WR for SQ for non RDMA Read operations */
> > + le32 max_send_sge;
> > + /* Maximum number of s/g elements per WR for RQ for non RDMA Read operations */
> > + le32 max_recv_sge;
> > + /* Maximum number of s/g per WR for RDMA Read operations */
> > + le32 max_sge_rd;
> > + /* Maximum size of Completion Queue (CQ) */
> > + le32 max_cqe;
>
> Need to specify the reason why we can't use the virtqueue size for the
> completion queue.
>
I think we can. Will remove it
> > + /* Maximum number of Memory Regions (MR) */
> > + le32 max_mr;
> > + /* Maximum number of Protection Domains (PD) */
> > + le32 max_pd;
> > + /* Maximum number of RDMA Read perations that can be outstanding per Queue Pair (QP) */
>
> I guess you mean "operations" here.
>
Yes.
> > + le32 max_qp_rd_atom;
> > + /* Maximum depth per QP for initiation of RDMA Read operations */
>
> The member has an "atom" suffix, does it mean "atomic read" or other?
>
It means the atomic operation which is unsupported now. I think we
need to remove it.
> > + le32 max_qp_init_rd_atom;
> > + /* Maximum number of Address Handles (AH) */
> > + le32 max_ah;
> > + /* Local CA ack delay */
> > + u8 local_ca_ack_delay;
> > + /* Padding */
> > + u8 padding[3];
> > + /* Reserved for future */
> > + le32 reserved[14];
> > +};
> > +\end{lstlisting}
> > +
> > +\item[VIRTIO_NET_CTRL_ROCE_QUERY_PORT] Query the attributes of port.
> > + No command-specific-data;
> > + the ack-specific-data is \field{struct virtio_rdma_ack_query_port}.
> > +
> > +\begin{lstlisting}
> > +struct virtio_rdma_ack_query_port {
> > + /* Length of source Global Identifier (GID) table */
> > + le32 gid_tbl_len;
> > + /* Maximum message size */
> > + le32 max_msg_sz;
>
> I guess this is for both read/write/send/receive? And is 4GB
> sufficient for the future?
>
Now this follows the definition in linux kernel and IB Spec. If we
need to extend it in future, we can add a new field max_msg_sz64?
> > + /* Reserved for future */
> > + le32 reserved[6];
> > +};
> > +\end{lstlisting}
> > +
> > +\item[VIRTIO_NET_CTRL_ROCE_CREATE_CQ] Create a Completion Queue (CQ).
> > + The command-specific-data is \field{struct virtio_rdma_cmd_create_cq};
> > + the ack-specific-data is \field{struct virtio_rdma_ack_create_cq}.
> > +
> > +\begin{lstlisting}
> > +struct virtio_rdma_cmd_create_cq {
> > + /* Size of CQ */
> > + le32 cqe;
> > +};
> > +
> > +struct virtio_rdma_ack_create_cq {
> > + /* The index of CQ */
> > + le32 cqn;
> > +};
> > +\end{lstlisting}
> > +
> > +\item[VIRTIO_NET_CTRL_ROCE_DESTROY_CQ] Destroy a Completion Queue.
> > + The command-specific-data is \field{struct virtio_rdma_cmd_destroy_cq};
> > + no ack-specific-data.
> > +
> > +\begin{lstlisting}
> > +struct virtio_rdma_cmd_destroy_cq {
> > + /* The index of CQ */
> > + le32 cqn;
> > +};
> > +\end{lstlisting}
> > +
> > +\item[VIRTIO_NET_CTRL_ROCE_CREATE_PD] Create a Protection Domain (PD).
> > + No command-specific-data;
> > + the ack-specific-data is \field{struct virtio_rdma_ack_create_pd}.
> > +
> > +\begin{lstlisting}
> > +struct virtio_rdma_ack_create_pd {
> > + /* The handle of PD */
> > + le32 pdn;
> > +};
> > +\end{lstlisting}
>
> Can this command always succeed? I meant is there a limit of the total
> number of PDs that a single ROCE device can support?
>
Yes, we have max_pd field in structure virtio_rdma_ack_query_device.
> > +
> > +\item[VIRTIO_NET_CTRL_ROCE_DESTORY_PD] Destroy a Protection Domain.
> > + The command-specific-data is \field{virtio_rdma_cmd_destroy_pd};
> > + no ack-specific-data.
> > +
> > +\begin{lstlisting}
> > +struct virtio_rdma_cmd_destroy_pd {
> > + /* The handle of PD */
> > + le32 pdn;
> > +};
> > +\end{lstlisting}
> > +
> > +\item[VIRTIO_NET_CTRL_ROCE_GET_DMA_MR] Get the DMA Memory Region (MR).
> > + associated with one protection domain.
>
> I wonder what's the difference between VIRTIO_NET_CTRL_ROCE_GET_DMA_MR
> and USR_MR. Can we unify them?
>
We should pass some address for USER_MR. I think we can unify them if we want.
> > + The command-specific-data is \field{virtio_rdma_cmd_get_dma_mr};
> > + the ack-specific-data is \field{virtio_rdma_ack_get_dma_mr}.
> > +
> > +\begin{lstlisting}
> > +enum virtio_ib_access_flags {
> > + VIRTIO_IB_ACCESS_LOCAL_WRITE = (1 << 0),
>
> Is LOCAL_READ implied to work always?
>
Yes, the LOCAL_READ is always supported.
> > + VIRTIO_IB_ACCESS_REMOTE_WRITE = (1 << 1),
> > + VIRTIO_IB_ACCESS_REMOTE_READ = (1 << 2),
> > +};
> > +
> > +struct virtio_rdma_cmd_get_dma_mr {
> > + /* The handle of PD which the MR associated with */
> > + le32 pdn;
> > + /* MR's protection attributes, enum virtio_ib_access_flags */
> > + le32 access_flags;
> > +};
> > +
> > +struct virtio_rdma_ack_get_dma_mr {
> > + /* The handle of MR */
> > + le32 mrn;
> > + /* MR's local access key */
> > + le32 lkey;
> > + /* MR's remote access key */
> > + le32 rkey;
> > +};
> > +\end{lstlisting}
> > +
> > +\item[VIRTIO_NET_CTRL_ROCE_REG_USER_MR] Register a user Memory Region
> > + associated with one Protection Domain.
> > + The command-specific-data is \field{virtio_rdma_cmd_reg_user_mr};
> > + the ack-specific-data is \field{virtio_rdma_ack_reg_user_mr}.
> > +
> > +\begin{lstlisting}
> > +struct virtio_rdma_cmd_reg_user_mr {
> > + /* The handle of PD which the MR associated with */
> > + le32 pdn;
> > + /* MR's protection attributes, enum virtio_ib_access_flags */
> > + le32 access_flags;
> > + /* Starting virtual address of MR */
> > + le64 virt_addr;
>
> I guess this is actually the I/O virtual address and the device is in
> charge of translate it to the page arrays below?
>
Yes, this address is specified by userspace, which can be a virtual
address or not.
> > + /* Length of MR */
> > + le64 length;
> > + /* Size of the below page array */
> > + le32 npages;
> > + /* Padding */
> > + le32 padding;
> > + /* Array to store physical address of each page in MR */
> > + le64 pages[];
>
> How do device know the size of a page?
>
We have npages field in this struture.
> > +};
>
> I believe this command can fail, we need to describe the error conditions.
>
OK.
> > +
> > +struct virtio_rdma_ack_reg_user_mr {
> > + /* The handle of MR */
> > + le32 mrn;
> > + /* MR's local access key */
> > + le32 lkey;
> > + /* MR's remote access key */
> > + le32 rkey;
> > +};
> > +\end{lstlisting}
> > +
> > +\item[VIRTIO_NET_CTRL_ROCE_DEREG_MR] De-register a Memory Region.
> > + The command-specific-data is \field{virtio_rdma_cmd_dereg_mr};
> > + no ack-specific-data.
> > +
> > +\begin{lstlisting}
> > +struct virtio_rdma_cmd_dereg_mr {
> > + /* The handle of MR */
> > + le32 mrn;
> > +};
> > +\end{lstlisting}
> > +
> > +\item[VIRTIO_NET_CTRL_ROCE_CREATE_QP] Create a Queue Pair (Send Queue and Receive Queue).
> > + The command-specific-data is \field{virtio_rdma_cmd_create_qp};
> > + the ack-specific-data is \field{virtio_rdma_ack_create_qp}.
> > +
> > +\begin{lstlisting}
> > +struct virtio_rdma_qp_cap {
> > + /* Maximum number of outstanding WRs in SQ */
> > + le32 max_send_wr;
> > + /* Maximum number of outstanding WRs in RQ */
> > + le32 max_recv_wr;
> > + /* Maximum number of s/g elements per WR in SQ */
> > + le32 max_send_sge;
> > + /* Maximum number of s/g elements per WR in RQ */
> > + le32 max_recv_sge;
> > + /* Maximum number of data (bytes) that can be posted inline to SQ */
> > + le32 max_inline_data;
> > + /* Padding */
> > + le32 padding;
> > +};
> > +
> > +struct virtio_rdma_cmd_create_qp {
> > + /* The handle of PD which the QP associated with */
> > + le32 pdn;
> > +#define VIRTIO_IB_QPT_SMI 0
> > +#define VIRTIO_IB_QPT_GSI 1
> > +#define VIRTIO_IB_QPT_RC 2
> > +#define VIRTIO_IB_QPT_UC 3
> > +#define VIRTIO_IB_QPT_UD 4
> > + /* QP's type */
> > + u8 qp_type;
> > + /* If set, each WR submitted to the SQ generates a completion entry */
> > + u8 sq_sig_all;
> > + /* Padding */
> > + u8 padding[2];
> > + /* The index of CQ which the SQ associated with */
> > + le32 send_cqn;
> > + /* The index of CQ which the RQ associated with */
> > + le32 recv_cqn;
> > + /* QP's capabilities */
> > + struct virtio_rdma_qp_cap cap;
> > + /* Reserved for future */
> > + le32 reserved[4];
> > +};
> > +
> > +struct virtio_rdma_ack_create_qp {
> > + /* The index of QP */
> > + le32 qpn;
> > +};
> > +\end{lstlisting}
> > +
> > +\item[VIRTIO_NET_CTRL_ROCE_MODIFY_QP] Modify the attributes of a Queue Pair.
> > + The command-specific-data is \field{virtio_rdma_cmd_modify_qp};
> > + no ack-specific-data.
> > +
> > +\begin{lstlisting}
> > +struct virtio_rdma_global_route {
> > + /* Destination GID or MGID */
> > + u8 dgid[16];
> > + /* Flow label */
> > + le32 flow_label;
> > + /* Source GID index */
> > + u8 sgid_index;
> > + /* Hop limit */
> > + u8 hop_limit;
> > + /* Traffic class */
> > + u8 traffic_class;
> > + /* Padding */
> > + u8 padding;
> > +};
> > +
> > +struct virtio_rdma_ah_attr {
> > + /* Global Routing Header (GRH) attributes */
> > + virtio_rdma_global_route grh;
> > + /* Destination MAC address */
> > + u8 dmac[6];
> > + /* Reserved for future */
> > + u8 reserved[10];
> > +};
> > +
> > +enum virtio_ib_qp_attr_mask {
> > + VIRTIO_IB_QP_STATE = (1 << 0),
> > + VIRTIO_IB_QP_CUR_STATE = (1 << 1),
> > + VIRTIO_IB_QP_ACCESS_FLAGS = (1 << 2),
> > + VIRTIO_IB_QP_QKEY = (1 << 3),
> > + VIRTIO_IB_QP_AV = (1 << 4),
> > + VIRTIO_IB_QP_PATH_MTU = (1 << 5),
> > + VIRTIO_IB_QP_TIMEOUT = (1 << 6),
> > + VIRTIO_IB_QP_RETRY_CNT = (1 << 7),
> > + VIRTIO_IB_QP_RNR_RETRY = (1 << 8),
> > + VIRTIO_IB_QP_RQ_PSN = (1 << 9),
> > + VIRTIO_IB_QP_MAX_QP_RD_ATOMIC = (1 << 10),
> > + VIRTIO_IB_QP_MIN_RNR_TIMER = (1 << 11),
> > + VIRTIO_IB_QP_SQ_PSN = (1 << 12),
> > + VIRTIO_IB_QP_MAX_DEST_RD_ATOMIC = (1 << 13),
> > + VIRTIO_IB_QP_CAP = (1 << 14),
> > + VIRTIO_IB_QP_DEST_QPN = (1 << 15),
> > + VIRTIO_IB_QP_RATE_LIMIT = (1 << 16),
> > +};
>
> Do we need to explain the above error codes? Or it's simply a map from IB spec?
>
Yes, it's defined in IB spec. But we can add some comments for them too.
> > +
> > +enum virtio_ib_qp_state {
> > + VIRTIO_IB_QPS_RESET,
> > + VIRTIO_IB_QPS_INIT,
> > + VIRTIO_IB_QPS_RTR,
> > + VIRTIO_IB_QPS_RTS,
> > + VIRTIO_IB_QPS_SQD,
> > + VIRTIO_IB_QPS_SQE,
> > + VIRTIO_IB_QPS_ERR
> > +};
> > +
> > +enum virtio_ib_mtu {
> > + VIRTIO_IB_MTU_256 = 1,
> > + VIRTIO_IB_MTU_512 = 2,
> > + VIRTIO_IB_MTU_1024 = 3,
> > + VIRTIO_IB_MTU_2048 = 4,
> > + VIRTIO_IB_MTU_4096 = 5
> > +};
> > +
> > +struct virtio_rdma_cmd_modify_qp {
> > + /* The index of QP */
> > + le32 qpn;
> > + /* The mask of attributes needs to be modified, enum virtio_ib_qp_attr_mask */
> > + le32 attr_mask;
> > + /* Move the QP to this state, enum virtio_ib_qp_state */
> > + u8 qp_state;
> > + /* Current QP state, enum virtio_ib_qp_state */
> > + u8 cur_qp_state;
> > + /* Path MTU (valid only for RC/UC QPs), enum virtio_ib_mtu */
> > + u8 path_mtu;
> > + /* Number of outstanding RDMA Read operations on destination QP (valid only for RC QPs) */
> > + u8 max_rd_atomic;
> > + /* Number of responder resources for handling incoming RDMA Read operations (valid only for RC QPs) */
> > + u8 max_dest_rd_atomic;
> > + /* Minimum RNR (Receiver Not Ready) NAK timer (valid only for RC QPs) */
> > + u8 min_rnr_timer;
> > + /* Local ack timeout (valid only for RC QPs) */
> > + u8 timeout;
> > + /* Retry count (valid only for RC QPs) */
> > + u8 retry_cnt;
> > + /* RNR retry (valid only for RC QPs) */
> > + u8 rnr_retry;
> > + /* Padding */
> > + u8 padding[7];
> > + /* Q_Key for the QP (valid only for UD QPs) */
> > + le32 qkey;
> > + /* PSN for RQ (valid only for RC/UC QPs) */
> > + le32 rq_psn;
> > + /* PSN for SQ */
> > + le32 sq_psn;
> > + /* Destination QP number (valid only for RC/UC QPs) */
> > + le32 dest_qp_num;
> > + /* Mask of enabled remote access operations (valid only for RC/UC QPs), enum virtio_ib_access_flags */
> > + le32 qp_access_flags;
> > + /* Rate limit in kbps for packet pacing */
> > + le32 rate_limit;
> > + /* QP capabilities */
> > + struct virtio_rdma_qp_cap cap;
> > + /* Address Vector (valid only for RC/UC QPs) */
> > + struct virtio_rdma_ah_attr ah_attr;
> > + /* Reserved for future */
> > + le32 reserved[4];
> > +};
> > +\end{lstlisting}
> > +
> > +\item[VIRTIO_NET_CTRL_ROCE_QUERY_QP] Query the attributes of a Queue Pair.
> > + The command-specific-data is \field{virtio_rdma_cmd_query_qp};
> > + the ack-specific-data is \field{virtio_rdma_ack_query_qp}.
> > +
> > +\begin{lstlisting}
> > +struct virtio_rdma_cmd_query_qp {
> > + /* The index of QP */
> > + le32 qpn;
> > + /* The mask of attributes need to be queried, enum virtio_ib_qp_attr_mask */
> > + le32 attr_mask;
> > +};
> > +
> > +struct virtio_rdma_ack_query_qp {
>
> Any chance to unify this with virtio_rdma_cmd_modify_qp?
>
It would be a little confusing since some states is only used by modify_qp.
> > + /* Move the QP to this state, enum virtio_ib_qp_state */
> > + u8 qp_state;
> > + /* Path MTU (valid only for RC/UC QPs), enum virtio_ib_mtu */
> > + u8 path_mtu;
> > + /* Is the SQ draining */
> > + u8 sq_draining;
> > + /* Number of outstanding RDMA read operations on destination QP (valid only for RC QPs) */
> > + u8 max_rd_atomic;
> > + /* Number of responder resources for handling incoming RDMA read operations (valid only for RC QPs) */
> > + u8 max_dest_rd_atomic;
> > + /* Minimum RNR NAK timer (valid only for RC QPs) */
> > + u8 min_rnr_timer;
> > + /* Local ack timeout (valid only for RC QPs) */
> > + u8 timeout;
> > + /* Retry count (valid only for RC QPs) */
> > + u8 retry_cnt;
> > + /* RNR retry (valid only for RC QPs) */
> > + u8 rnr_retry;
> > + /* Padding */
> > + u8 padding[7];
> > + /* Q_Key for the QP (valid only for UD QPs) */
> > + le32 qkey;
> > + /* PSN for RQ (valid only for RC/UC QPs) */
> > + le32 rq_psn;
> > + /* PSN for SQ */
> > + le32 sq_psn;
> > + /* Destination QP number (valid only for RC/UC QPs) */
> > + le32 dest_qp_num;
> > + /* Mask of enabled remote access operations (valid only for RC/UC QPs), enum virtio_ib_access_flags */
> > + le32 qp_access_flags;
> > + /* Rate limit in kbps for packet pacing */
> > + le32 rate_limit;
> > + /* QP capabilities */
> > + struct virtio_rdma_qp_cap cap;
> > + /* Address Vector (valid only for RC/UC QPs) */
> > + struct virtio_rdma_ah_attr ah_attr;
> > + /* Reserved for future */
> > + le32 reserved[4];
> > +};
> > +\end{lstlisting}
> > +
> > +\item[VIRTIO_NET_CTRL_ROCE_DESTROY_QP] Destroy a Queue Pair.
> > + The command-specific-data is \field{virtio_rdma_cmd_destroy_qp};
> > + no ack-specific-data.
>
> What happen to the pending requests? Will the device wait for the
> completion or not?
>
It should be discarded according to IB spec.
> > +
> > +\begin{lstlisting}
> > +struct virtio_rdma_cmd_destroy_qp {
> > + /* The index of QP */
> > + le32 qpn;
> > +};
> > +\end{lstlisting}
> > +
> > +\item[VIRTIO_NET_CTRL_ROCE_CREATE_AH] Create a Address Handle (AH).
> > + The command-specific-data is \field{virtio_rdma_cmd_create_ah};
> > + the ack-specific-data is \field{virtio_rdma_ack_create_ah}.
> > +
> > +\begin{lstlisting}
> > +struct virtio_rdma_cmd_create_ah {
> > + /* The handle of PD which the AH associated with */
> > + le32 pdn;
> > + /* Padding */
> > + le32 padding;
> > + /* Address Vector */
> > + struct virtio_rdma_ah_attr ah_attr;
> > +};
> > +
> > +struct virtio_rdma_ack_create_ah {
> > + /* The address handle */
> > + le32 ah;
> > +};
> > +\end{lstlisting}
> > +
> > +\item[VIRTIO_NET_CTRL_ROCE_DESTROY_AH] Destroy a Address Handle.
> > + The command-specific-data is \field{virtio_rdma_cmd_destroy_ah};
> > + no ack-specific-data.
> > +
> > +\begin{lstlisting}
> > +struct virtio_rdma_cmd_destroy_ah {
> > + /* The handle of PD which the AH associated with */
> > + le32 pdn;
> > + /* The address handle */
> > + le32 ah;
> > +};
> > +\end{lstlisting}
> > +
> > +\item[VIRTIO_NET_CTRL_ROCE_ADD_GID] Add a Global Identifier (GID).
> > + The command-specific-data is \field{virtio_rdma_cmd_add_gid};
> > + no ack-specific-data.
> > +
> > +\begin{lstlisting}
> > +struct virtio_rdma_cmd_add_gid {
> > + /* Index of GID */
> > + le16 index;
> > + /* Padding */
> > + le16 padding[3];
> > + /* GID to be added */
> > + u8 gid[16];
> > +};
> > +\end{lstlisting}
> > +
> > +\item[VIRTIO_NET_CTRL_ROCE_DEL_GID] Delete a Global Identifier.
> > + The command-specific-data is \field{virtio_rdma_cmd_del_gid};
> > + no ack-specific-data.
> > +
> > +\begin{lstlisting}
> > +struct virtio_rdma_cmd_del_gid {
> > + /* Index of GID */
> > + le16 index;
> > +};
> > +\end{lstlisting}
> > +
> > +\item[VIRTIO_NET_CTRL_ROCE_REQ_NOTIFY_CQ] Request a completion notification
> > + on a Completion Queue.
> > + The command-specific-data is \field{virtio_rdma_cmd_req_notify};
> > + no ack-specific-data.
> > +
> > +\begin{lstlisting}
> > +struct virtio_rdma_cmd_req_notify {
> > + /* The index of CQ */
> > + le32 cqn;
> > +#define VIRTIO_IB_NOTIFY_SOLICITED (1 << 0)
> > +#define VIRTIO_IB_NOTIFY_NEXT_COMPLETION (1 << 1)
>
> Need to describe the differences on those two flags.
>
OK.
> > + /* Notify flags */
> > + le32 flags;
> > +};
> > +\end{lstlisting}
> > +
> > +\end{description}
> > +
> > +\drivernormative{\subparagraph}{RoCE Configuration}{Device Types / Network Device / Device Operation / Control Virtqueue / RoCE Configuration}
> > +
> > +A driver MUST initialize the completion virtqueue and fill it with
> > +enough entries after command VIRTIO_NET_CTRL_ROCE_CREATE_CQ is
> > +successfully executed.
> > +
> > +A driver MUST reset the completion virtqueue after
>
> How to do the reset? Do you mean driver need to reset the indices?
>
Yes, something like avail_idx, used_idx.
> > +command VIRTIO_NET_CTRL_ROCE_DESTROY_CQ is successfully executed.
> > +
> > +A driver MUST initialize the send virtqueue and receive virtqueue after
> > +command VIRTIO_NET_CTRL_ROCE_CREATE_QP is successfully executed.
> > +
> > +A driver MUST reset the send virtqueue and receive virtqueue after
> > +command VIRTIO_NET_CTRL_ROCE_DESTROY_QP is successfully executed.
> >
> > \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device
> > Types / Network Device / Legacy Interface: Framing Requirements}
> > @@ -4496,6 +5063,289 @@ \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device
> > See \ref{sec:Basic
> > Facilities of a Virtio Device / Virtqueues / Message Framing}.
> >
> > +\subsubsection{RoCE Support}\label{sec:Device Types / Network Device / Device Operation / RoCE Support}
> > +
> > +RDMA over Converged Ethernet (RoCE) is a network protocol that allows
> > +remote direct memory access (RDMA) over an Ethernet network. To support
> > +RoCE (if VIRTIO_NET_F_ROCE is negotiated), in addtion to the control
> > +virtqueue support mentioned in \ref{sec:Device Types / Network Device /
> > +Device Operation / Control Virtqueue / RoCE Configuration}, multiple
> > +types of virtqueues including send virtqueue, receive virtqueue and
> > +completion virtqueue are introduced.
> > +
> > +The send virtqueue contains elements that describe the data to be
> > +transmitted.
> > +
> > +Requests (device-readable) have the following format:
> > +
> > +\begin{lstlisting}
> > +enum virtio_ib_wr_opcode {
> > + VIRTIO_IB_WR_RDMA_WRITE,
> > + VIRTIO_IB_WR_RDMA_WRITE_WITH_IMM,
> > + VIRTIO_IB_WR_SEND,
> > + VIRTIO_IB_WR_SEND_WITH_IMM,
> > + VIRTIO_IB_WR_RDMA_READ,
> > +};
> > +
> > +struct virtio_rdma_sge {
> > + le64 addr;
> > + le32 length;
> > + le32 lkey;
> > +};
> > +
> > +struct virtio_rdma_sq_req {
> > + /* User defined WR ID */
> > + le64 wr_id;
> > + /* WR opcode, enum virtio_ib_wr_opcode */
> > + u8 opcode;
> > +#define VIRTIO_IB_SEND_FENCE (1 << 0)
> > +#define VIRTIO_IB_SEND_SIGNALED (1 << 1)
> > +#define VIRTIO_IB_SEND_SOLICITED (1 << 2)
> > +#define VIRTIO_IB_SEND_INLINE (1 << 3)
> > + /* Flags of the WR properties */
> > + u8 send_flags;
> > + /* Padding */
> > + le16 padding;
> > + /* Immediate data (in network byte order) to send */
> > + le32 imm_data;
> > + union {
> > + struct {
> > + /* Start address of remote memory buffer */
> > + le64 remote_addr;
> > + /* Key of the remote MR */
> > + le32 rkey;
> > + } rdma;
> > + struct {
> > + /* Index of the destination QP */
> > + le32 remote_qpn;
> > + /* Q_Key of the destination QP */
> > + le32 remote_qkey;
> > + /* Address Handle */
> > + le32 ah;
> > + } ud;
> > + /* Reserved for future */
> > + le64 reserved[4];
> > + };
> > + /* Inline data */
> > + u8 inline_data[512];
> > + union {
> > + /* Length of sg_list */
> > + le32 num_sge;
> > + /* Length of inline data */
> > + le16 inline_len;
> > + };
> > + /* Reserved for future */
> > + le32 reserved2[3];
> > + /* Scatter/gather list */
> > + struct virtio_rdma_sge sg_list[];
> > +};
> > +\end{lstlisting}
> > +
> > +The receive virtqueue contains elements that describe where to place incoming data.
> > +
> > +Requests (device-readable) have the following format:
> > +
> > +\begin{lstlisting}
> > +struct virtio_rdma_rq_req {
> > + /* User defined WR ID */
> > + le64 wr_id;
> > + /* Length of sg_list */
> > + le32 num_sge;
> > + /* Reserved for future */
> > + le32 reserved[3];
> > + /* Scatter/gather list */
> > + struct virtio_rdma_sge sg_list[];
> > +};
> > +\end{lstlisting}
> > +
> > +The completion virtqueue is used to notify the completion of requests in
> > +send virtqueue or receive virtqueue.
> > +
> > +Requests (device-writable) have the following format:
> > +
> > +\begin{lstlisting}
> > +enum virtio_ib_wc_opcode {
> > + VIRTIO_IB_WC_SEND,
> > + VIRTIO_IB_WC_RDMA_WRITE,
> > + VIRTIO_IB_WC_RDMA_READ,
> > + VIRTIO_IB_WC_RECV,
> > + VIRTIO_IB_WC_RECV_RDMA_WITH_IMM,
> > +};
> > +
> > +enum virtio_ib_wc_status {
> > + /* Operation completed successfully */
> > + VIRTIO_IB_WC_SUCCESS,
> > + /* Local Length Error */
> > + VIRTIO_IB_WC_LOC_LEN_ERR,
> > + /* Local QP Operation Error */
> > + VIRTIO_IB_WC_LOC_QP_OP_ERR,
> > + /* Local Protection Error */
> > + VIRTIO_IB_WC_LOC_PROT_ERR,
> > + /* Work Request Flushed Error */
> > + VIRTIO_IB_WC_WR_FLUSH_ERR,
> > + /* Bad Response Error */
> > + VIRTIO_IB_WC_BAD_RESP_ERR,
> > + /* Local Access Error */
> > + VIRTIO_IB_WC_LOC_ACCESS_ERR,
> > + /* Remote Invalid Request Error */
> > + VIRTIO_IB_WC_REM_INV_REQ_ERR,
> > + /* Remote Access Error */
> > + VIRTIO_IB_WC_REM_ACCESS_ERR,
> > + /* Remote Operation Error */
> > + VIRTIO_IB_WC_REM_OP_ERR,
> > + /* Transport Retry Counter Exceeded */
> > + VIRTIO_IB_WC_RETRY_EXC_ERR,
> > + /* RNR Retry Counter Exceeded */
> > + VIRTIO_IB_WC_RNR_RETRY_EXC_ERR,
> > + /* Remote Aborted Error */
> > + VIRTIO_IB_WC_REM_ABORT_ERR,
> > + /* Fatal Error */
> > + VIRTIO_IB_WC_FATAL_ERR,
> > + /* Response Timeout Error */
> > + VIRTIO_IB_WC_RESP_TIMEOUT_ERR,
> > + /* General Error */
> > + VIRTIO_IB_WC_GENERAL_ERR
> > +};
> > +
> > +struct virtio_rdma_cq_req {
> > + /* User defined WR ID */
> > + le64 wr_id;
> > + /* Work completion status, enum virtio_ib_wc_status */
> > + u8 status;
> > + /* WR opcode, enum virtio_ib_wc_opcode */
> > + u8 opcode;
> > + /* Padding */
> > + le16 padding;
> > + /* Vendor error */
> > + le32 vendor_err;
> > + /* Number of bytes transferred */
> > + le32 byte_len;
> > + /* Immediate data (in network byte order) to send */
> > + le32 imm_data;
> > + /* Local QP number of completed WR */
> > + le32 qp_num;
> > + /* Source QP number (remote QP number) of completed WR (valid only for UD QPs) */
> > + le32 src_qp;
> > +#define VIRTIO_IB_WC_GRH (1 << 0)
> > +#define VIRTIO_IB_WC_WITH_IMM (1 << 1)
> > + /* Work completion flag */
> > + le32 wc_flags;
> > + /* Reserved for future */
> > + le32 reserved[3];
> > +};
> > +\end{lstlisting}
> > +
> > +\paragraph{Send Operation}\label{sec:Device Types / Network Device / Device Operation / RoCE Support / Send Operation}
> > +
> > +The send operation allows us to send data to a remote QPâs Receive Queue.
> > +The receiver MUST have previously posted a receive buffer to receive the data.
>
> "MUST" keyword must belong to the normative section.
>
OK.
> > +
> > +To do a send operation, a request with \field{opcode} set to
> > +VIRTIO_IB_WR_SEND or VIRTIO_IB_WR_SEND_WITH_IMM MUST be posted to the Send
> > +Queue as one output descriptor and the device is notified of the new entry.
> > +
> > +\drivernormative{\subparagraph}{Send Operation}{Device Types / Network Device / Device Operation / RoCE Support / Send Operation}
> > +
> > +If VIRTIO_IB_SEND_INLINE is set in \field{send_flags}, the driver MUST fill
> > +send buffer into \field{inline_data} field and set \field{inline_len} to the
> > +length of the buffer. Otherwise, the driver MUST fill \field{sg_list} to
> > +describe the buffer.
> > +
> > +\devicenormative{\subparagraph}{Send Operation}{Device Types / Network Device / Device Operation / RoCE Support / Send Operation}
> > +
> > +If \field{opcode} is not set to VIRTIO_IB_WR_SEND_WITH_IMM, the device MUST
> > +ignore \field{imm_data}.
> > +
> > +If the QP type is UD, the device MUST validate \field{ud.ah}.
> > +
> > +If VIRTIO_IB_SEND_INLINE is not set in \field{send_flags}, the device MUST
> > +validate the \field{addr}, \field{length} and \field{lkey} in \field{sg_list}.
> > +
> > +\paragraph{Receive Operation}\label{sec:Device Types / Network Device / Device Operation / RoCE Support / Receive Operation}
> > +
> > +The receive operation allows us to receive data from remote QP.
> > +It's the corresponding operation to a send operation.
> > +
> > +To do a receive operation, a request MUST be posted to the Receive
> > +Queue as one output descriptor and the device is notified of the new entry.
> > +
>
> I think we probably need to be more verbose as what has been done for
> virtio-net.
>
> That is, describe what need to be filled in virtio_rdma_rq_req in
> details. (And do this for other operation as well)
>
OK.
>
> > +\drivernormative{\subparagraph}{Receive Operation}{Device Types / Network Device / Device Operation / RoCE Support / Receive Operation}
> > +
> > +The driver MUST fill \field{sg_list} to describe the receive buffer.
> > +
> > +\devicenormative{\subparagraph}{Receive Operation}{Device Types / Network Device / Device Operation / RoCE Support / Receive Operation}
> > +
> > +The device MUST validate the \field{addr}, \field{length} and \field{lkey}
> > +in \field{sg_list}.
> > +
> > +\paragraph{Write Operation}\label{sec:Device Types / Network Device / Device Operation / RoCE Support / Write Operation}
> > +
> > +The write operation allows us to write data to the local memory buffer
> > +in remote side with no notification. The remote side wouldn't be aware
> > +that this operation being done.
> > +
> > +To do a write operation, a request with \field{opcode} set to
> > +VIRTIO_IB_WR_RDMA_WRITE or VIRTIO_IB_WR_RDMA_WRITE_WITH_IMM MUST be
> > +posted to the Send Queue as one output descriptor and the device is
> > +notified of the new entry.
> > +
> > +\drivernormative{\subparagraph}{Write Operation}{Device Types / Network Device / Device Operation / RoCE Support / Write Operation}
> > +
> > +The driver MUST fill \field{sg_list} to describe the write buffer.
>
> So sg is a must even if the driver want to use imm?
>
Looks like not. I will fix it.
> > +
> > +The driver MUST fill \field{rdma.remote_addr} and \field{rdma.rkey} to
> > +identify the remote buffer.
> > +
> > +\devicenormative{\subparagraph}{Write Operation}{Device Types / Network Device / Device Operation / RoCE Support / Write Operation}
> > +
> > +If \field{opcode} is not set to VIRTIO_IB_WR_RDMA_WRITE_WITH_IMM, the device
> > +MUST ignore \field{imm_data}.
> > +
> > +The device MUST validate the \field{addr}, \field{length} and \field{lkey}
> > +in \field{sg_list}.
> > +
> > +\paragraph{Read Operation}\label{sec:Device Types / Network Device / Device Operation / RoCE Support / Read Operation}
> > +
> > +The read operation allows us to read data from the local memory buffer
> > +in remote side with no notification. The remote side wouldn't be aware
> > +that this operation being done.
> > +
> > +To do a read operation, a request with \field{opcode} set to
> > +VIRTIO_IB_WR_RDMA_READ MUST be posted to the Send Queue as one output
> > +descriptor and the device is notified of the new entry.
> > +
> > +\drivernormative{\subparagraph}{Read Operation}{Device Types / Network Device / Device Operation / RoCE Support / Read Operation}
> > +
> > +The driver MUST fill \field{sg_list} to describe the read buffer.
> > +
> > +The driver MUST fill \field{rdma.remote_addr} and \field{rdma.rkey} to
> > +identify the remote buffer.
> > +
> > +\devicenormative{\subparagraph}{Read Operation}{Device Types / Network Device / Device Operation / RoCE Support / Read Operation}
> > +
> > +The device MUST validate the \field{addr}, \field{length} and \field{lkey}
> > +in \field{sg_list}.
> > +
> > +\paragraph{Completion Notification}\label{sec:Device Types / Network Device / Device Operation / RoCE Support / Completion Notification}
> > +
> > +After above operation is completed, a completion notification MUST
> > +be triggered by the device.
>
> For "completion notification", do you mean the virtqueue notification
> of cq or the making the buffer than contains cqe used?
>
Both? Making the buffer that contains cqe used and notify the virtqueue.
> > To achieve that, the device MUST consume
> > +an entry of the Completion Queue associated with the Send Queue/Receive
> > +Queue which the operation belongs to.
> > +
> > +\drivernormative{\subparagraph}{Completion Notification}{Device Types / Network Device / Device Operation / RoCE Support / Completion Notification}
> > +
> > +The driver MUST fill the Completion Queue with enough entries previously.
>
> What do you mean by "previously"? What happens if there's no sufficient cqe?
>
We need to fill the Completion Queue in advance. Otherwise, the driver
would not get completion notification after some operation is
completed.
Thanks,
Yongji
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]