[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]
Subject: Re: [RFC v2] virtio-net: Add RoCE (RDMA over Converged Ethernet) support
On Wed, May 11, 2022 at 5:59 PM Xie Yongji <xieyongji@bytedance.com> wrote:
>
> Hi all,
>
Not very familiar with ROCE, try to give some comments from general
virtio level.
> This RFC aims to introduce our recent work on enabling RoCE support
> for virtio-net device.
We need to clarify the version of ROCE, is it ROCEv2 or not?
>
> To support RoCE, three types of virtqueues including RDMA send virtqueue,
> RDMA receive virtqueue and RDMA completion virtqueue are introduced.
> And control virtqueue is reused to support the RDMA control messages.
>
> Now we support some basic RDMA semantics such as send/receive
> and read/write operation.
It would be better to explain the advantages of this over the existing
pvrdma approach. I guess one advantage is that using virtio makes it
easier to connect to a userspace dataplane through vDPA/vhost-user?
>
> To test with our demo:
>
> 1. Build Guest kernel [1] with config INFINIBAND_VIRTIO_RDMA
>
> 2. Build QEMU [2] with config VHOST_USER_RDMA
>
> 3. Build rdma-core [3]
>
> 4. Build and install DPDK (NOTE that we only tested on DPDK 20.11.3)
>
> 5. Build vhost-user-rdma [4]
>
> 6. Run vhost-user-rdma with command:
> $ ./vhost-user-rdma --vdev 'net_tap0' --lcore '1-3' -- -s '/tmp/vhost-rdma0'
>
> 7. Run qemu with command:
> $ qemu-system-x86_64 -chardev socket,path=/tmp/vhost-rdma0,id=vrdma \
> -device vhost-user-rdma-pci,page-per-vq,chardev=vrdma ...
It would be better to give some performance numbers (or even compare
it with pvrdma).
>
> [1] https://github.com/bytedance/linux/tree/virtio-net-roce
> [2] https://github.com/bytedance/qemu/tree/vhost-user-rdma
> [3] https://github.com/YongjiXie/rdma-core/tree/virtio-rdma
> [4] https://github.com/YongjiXie/vhost-user-rdma
>
> We have already tested it with ibv_rc_pingpong, ibv_ud_pingpong and some
> others in rdma-core.
>
> TODO:
>
And we'd better consider the live migration support. Having a quick
glance, it looks to me trapping the cvq is sufficient?
> 1. Add support for Base Memory Management Extensions
>
> 2. Add support for atomic operation
>
> 3. Add support for SRQ
>
> 4. Add support for virtqueue resize
Note that this is already supported by the spec via virtqueue reset.
>
> 5. Add support for enabling/disabling virtqueue at runtime
I guess virtqueue reset could help in this case.
>
> Please review, thanks!
>
> V1 to V2:
> - Rework the implementation via extending virtio-net instead of
> introducing a new device type [Jason]
> - Add address handle support
>
> Signed-off-by: Xie Yongji <xieyongji@bytedance.com>
> Co-developed-by: Wei Junji <weijunji@bytedance.com>
> Signed-off-by: Wei Junji <weijunji@bytedance.com>
> ---
> content.tex | 858 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
> 1 file changed, 854 insertions(+), 4 deletions(-)
I wonder if there's some open-source ROCE transport device API that we
can re-use then we can just behave like a transport layer instead of
inventing new commands.
>
> diff --git a/content.tex b/content.tex
> index 7508dd1..646d82a 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -3008,7 +3008,10 @@ \section{Network Device}\label{sec:Device Types / Network Device}
> placed in one virtqueue for receiving packets, and outgoing
> packets are enqueued into another for transmission in that order.
> A third command queue is used to control advanced filtering
> -features.
> +features. And if RoCE (RDMA over Converged Ethernet) capability
> +is enabled, the virtio network device can also support transmitting
> +and receiving RDMA message through RDMA send virtqueue, RDMA receive
> +virtqueue and RDMA completion virtqueue.
>
> \subsection{Device ID}\label{sec:Device Types / Network Device / Device ID}
>
> @@ -3023,13 +3026,24 @@ \subsection{Virtqueues}\label{sec:Device Types / Network Device / Virtqueues}
> \item[2(N-1)] receiveqN
> \item[2(N-1)+1] transmitqN
> \item[2N] controlq
> +\item[2N+1] rdma_completeq1
> +\item[\ldots]
> +\item[2N+M] rdma_completeqM
> +\item[2N+M+1] rdma_transmitq1
> +\item[2N+M+2] rdma_receiveq1
> +\item[\ldots]
> +\item[2N+M+2L-1] rdma_transmitqL
> +\item[2N+M+2L] rdma_receiveqL
> \end{description}
>
> N=1 if neither VIRTIO_NET_F_MQ nor VIRTIO_NET_F_RSS are negotiated, otherwise N is set by
> - \field{max_virtqueue_pairs}.
> + \field{max_virtqueue_pairs}. M is set by \field{max_rdma_cqs} and L is set by
> + \field{max_rdma_qps}.
>
> controlq only exists if VIRTIO_NET_F_CTRL_VQ set.
>
> + rdma_completeq, rdma_transmitq and rdma_receiveq only exist if VIRTIO_NET_F_ROCE set
> +
> \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits}
>
> \begin{description}
> @@ -3084,6 +3098,9 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits
> \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
> channel.
>
> +\item[VIRTIO_NET_F_ROCE(55)] Device supports RoCE (RDMA over Converged Ethernet)
> + capability.
> +
> \item[VIRTIO_NET_F_HOST_USO (56)] Device can receive USO packets. Unlike UFO
> (fragmenting the packet) the USO splits large UDP packet
> to several segments when each of these smaller packets has UDP header.
> @@ -3129,6 +3146,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device
> \item[VIRTIO_NET_F_GUEST_ANNOUNCE] Requires VIRTIO_NET_F_CTRL_VQ.
> \item[VIRTIO_NET_F_MQ] Requires VIRTIO_NET_F_CTRL_VQ.
> \item[VIRTIO_NET_F_CTRL_MAC_ADDR] Requires VIRTIO_NET_F_CTRL_VQ.
> +\item[VIRTIO_NET_F_ROCE] Requires VIRTIO_NET_F_CTRL_VQ.
> \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
> \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
> \end{description}
> @@ -3190,6 +3208,8 @@ \subsection{Device configuration layout}\label{sec:Device Types / Network Device
> u8 rss_max_key_size;
> le16 rss_max_indirection_table_length;
> le32 supported_hash_types;
> + le32 max_rdma_qps;
> + le32 max_rdma_cps;
> };
> \end{lstlisting}
> The following field, \field{rss_max_key_size} only exists if VIRTIO_NET_F_RSS or VIRTIO_NET_F_HASH_REPORT is set.
> @@ -3204,11 +3224,23 @@ \subsection{Device configuration layout}\label{sec:Device Types / Network Device
> Field \field{supported_hash_types} contains the bitmask of supported hash types.
> See \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled hash types} for details of supported hash types.
>
> +Field \field{max_rdma_qps} only exists if VIRTIO_NET_F_ROCE is set.
> +It specifies the maximum number of queue pairs (send virtqueue and receive virtqueue) for RoCE usage.
> +
> +Field \field{max_rdma_cqs} only exists if VIRTIO_NET_F_ROCE is set.
> +It specifies the maximum number of completion virtqueue for RoCE usage.
> +
> \devicenormative{\subsubsection}{Device configuration layout}{Device Types / Network Device / Device configuration layout}
>
> The device MUST set \field{max_virtqueue_pairs} to between 1 and 0x8000 inclusive,
> if it offers VIRTIO_NET_F_MQ.
>
> +The device MUST set \field{max_rdma_qps} to between 1 an 16384 inclusive,
> +if it offers VIRTIO_NET_F_ROCE.
I wonder why 16384 is chosen here?
> +
> +The device MUST set \field{max_rdma_cqs} to between 1 an 16384 inclusive,
> +if it offers VIRTIO_NET_F_ROCE.
> +
> The device MUST set \field{mtu} to between 68 and 65535 inclusive,
> if it offers VIRTIO_NET_F_MTU.
>
> @@ -3306,6 +3338,12 @@ \subsection{Device Initialization}\label{sec:Device Types / Network Device / Dev
> \item If the VIRTIO_NET_F_CTRL_VQ feature bit is negotiated,
> identify the control virtqueue.
>
> +\item If the VIRTIO_NET_F_ROCE feature bit is negotiated,
> + identify the the RDMA completion virtqueues, up to max_rdma_cqs.
> +
> +\item If the VIRTIO_NET_F_ROCE feature bit is negotiated,
> + identify the the RDMA send and receive virtqueues, up to max_rdma_qps.
> +
> \item Fill the receive queues with buffers: see \ref{sec:Device Types / Network Device / Device Operation / Setting Up Receive Buffers}.
>
> \item Even with VIRTIO_NET_F_MQ, only receiveq1, transmitq1 and
> @@ -4007,6 +4045,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> u8 command;
> u8 command-specific-data[];
> u8 ack;
> + u8 ack-specific-data[];
> };
>
> /* ack values */
> @@ -4015,8 +4054,8 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> \end{lstlisting}
>
> The \field{class}, \field{command} and command-specific-data are set by the
> -driver, and the device sets the \field{ack} byte. There is little it can
> -do except issue a diagnostic if \field{ack} is not
> +driver, and the device sets the \field{ack} byte and ack-specific-data. There
> +is little it can do except issue a diagnostic if \field{ack} is not
> VIRTIO_NET_OK.
>
> \paragraph{Packet Receive Filtering}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Packet Receive Filtering}
> @@ -4463,6 +4502,534 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> according to the native endian of the guest rather than
> (necessarily when not using the legacy interface) little-endian.
>
> +\paragraph{RoCE Configuration}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / RoCE Configuration}
> +
> +If the driver negotiates the VIRTIO_NET_F_ROCE feature bit (depends on VIRTIO_NET_F_CTRL_VQ),
> +it can send control commands for RoCE usage. The following commands are defined now:
> +
> +\begin{lstlisting}
> +#define VIRTIO_NET_CTRL_ROCE 6
> + #define VIRTIO_NET_CTRL_ROCE_QUERY_DEVICE 0
> + #define VIRTIO_NET_CTRL_ROCE_QUERY_PORT 1
> + #define VIRTIO_NET_CTRL_ROCE_CREATE_CQ 2
> + #define VIRTIO_NET_CTRL_ROCE_DESTROY_CQ 3
> + #define VIRTIO_NET_CTRL_ROCE_CREATE_PD 4
> + #define VIRTIO_NET_CTRL_ROCE_DESTROY_PD 5
> + #define VIRTIO_NET_CTRL_ROCE_GET_DMA_MR 6
> + #define VIRTIO_NET_CTRL_ROCE_REG_USER_MR 7
> + #define VIRTIO_NET_CTRL_ROCE_DEREG_MR 8
> + #define VIRTIO_NET_CTRL_ROCE_CREATE_QP 9
> + #define VIRTIO_NET_CTRL_ROCE_MODIFY_QP 10
> + #define VIRTIO_NET_CTRL_ROCE_QUERY_QP 11
> + #define VIRTIO_NET_CTRL_ROCE_DESTROY_QP 12
> + #define VIRTIO_NET_CTRL_ROCE_CREATE_AH 13
> + #define VIRTIO_NET_CTRL_ROCE_DESTROY_AH 14
> + #define VIRTIO_NET_CTRL_ROCE_ADD_GID 15
> + #define VIRTIO_NET_CTRL_ROCE_DEL_GID 16
> + #define VIRTIO_NET_CTRL_ROCE_REQ_NOTIFY_CQ 17
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[VIRTIO_NET_CTRL_ROCE_QUERY_DEVICE] Query the attributes of device.
> + No command-specific-data;
> + the ack-specific-data is \field{struct virtio_rdma_ack_query_device}.
> +
> +\begin{lstlisting}
> +struct virtio_rdma_ack_query_device {
> +#define VIRTIO_IB_DEVICE_RC_RNR_NAK_GEN (1 << 0)
What's the meaning of this capability?
> + /* Capabilities mask */
> + le64 device_cap_flags;
Will this introduce a migration compatibility issue? E.g src and dst
have the same features but different capabilities.
> + /* Largest contiguous block that can be registered */
> + le64 max_mr_size;
> + /* Supported memory shift sizes */
> + le64 page_size_cap;
> + /* Hardware version */
> + le32 hw_ver;
What did "hardware version" mean? Is this something that is defined in
the IB spec?
> + /* Maximum number of outstanding Work Requests (WR) on Send Queue (SQ) and Receive Queue (RQ) */
> + le32 max_qp_wr;
Is this implied in the virtqueue size? If not, why?
> + /* Maximum number of scatter/gather (s/g) elements per WR for SQ for non RDMA Read operations */
> + le32 max_send_sge;
> + /* Maximum number of s/g elements per WR for RQ for non RDMA Read operations */
> + le32 max_recv_sge;
> + /* Maximum number of s/g per WR for RDMA Read operations */
> + le32 max_sge_rd;
> + /* Maximum size of Completion Queue (CQ) */
> + le32 max_cqe;
Need to specify the reason why we can't use the virtqueue size for the
completion queue.
> + /* Maximum number of Memory Regions (MR) */
> + le32 max_mr;
> + /* Maximum number of Protection Domains (PD) */
> + le32 max_pd;
> + /* Maximum number of RDMA Read perations that can be outstanding per Queue Pair (QP) */
I guess you mean "operations" here.
> + le32 max_qp_rd_atom;
> + /* Maximum depth per QP for initiation of RDMA Read operations */
The member has an "atom" suffix, does it mean "atomic read" or other?
> + le32 max_qp_init_rd_atom;
> + /* Maximum number of Address Handles (AH) */
> + le32 max_ah;
> + /* Local CA ack delay */
> + u8 local_ca_ack_delay;
> + /* Padding */
> + u8 padding[3];
> + /* Reserved for future */
> + le32 reserved[14];
> +};
> +\end{lstlisting}
> +
> +\item[VIRTIO_NET_CTRL_ROCE_QUERY_PORT] Query the attributes of port.
> + No command-specific-data;
> + the ack-specific-data is \field{struct virtio_rdma_ack_query_port}.
> +
> +\begin{lstlisting}
> +struct virtio_rdma_ack_query_port {
> + /* Length of source Global Identifier (GID) table */
> + le32 gid_tbl_len;
> + /* Maximum message size */
> + le32 max_msg_sz;
I guess this is for both read/write/send/receive? And is 4GB
sufficient for the future?
> + /* Reserved for future */
> + le32 reserved[6];
> +};
> +\end{lstlisting}
> +
> +\item[VIRTIO_NET_CTRL_ROCE_CREATE_CQ] Create a Completion Queue (CQ).
> + The command-specific-data is \field{struct virtio_rdma_cmd_create_cq};
> + the ack-specific-data is \field{struct virtio_rdma_ack_create_cq}.
> +
> +\begin{lstlisting}
> +struct virtio_rdma_cmd_create_cq {
> + /* Size of CQ */
> + le32 cqe;
> +};
> +
> +struct virtio_rdma_ack_create_cq {
> + /* The index of CQ */
> + le32 cqn;
> +};
> +\end{lstlisting}
> +
> +\item[VIRTIO_NET_CTRL_ROCE_DESTROY_CQ] Destroy a Completion Queue.
> + The command-specific-data is \field{struct virtio_rdma_cmd_destroy_cq};
> + no ack-specific-data.
> +
> +\begin{lstlisting}
> +struct virtio_rdma_cmd_destroy_cq {
> + /* The index of CQ */
> + le32 cqn;
> +};
> +\end{lstlisting}
> +
> +\item[VIRTIO_NET_CTRL_ROCE_CREATE_PD] Create a Protection Domain (PD).
> + No command-specific-data;
> + the ack-specific-data is \field{struct virtio_rdma_ack_create_pd}.
> +
> +\begin{lstlisting}
> +struct virtio_rdma_ack_create_pd {
> + /* The handle of PD */
> + le32 pdn;
> +};
> +\end{lstlisting}
Can this command always succeed? I meant is there a limit of the total
number of PDs that a single ROCE device can support?
> +
> +\item[VIRTIO_NET_CTRL_ROCE_DESTORY_PD] Destroy a Protection Domain.
> + The command-specific-data is \field{virtio_rdma_cmd_destroy_pd};
> + no ack-specific-data.
> +
> +\begin{lstlisting}
> +struct virtio_rdma_cmd_destroy_pd {
> + /* The handle of PD */
> + le32 pdn;
> +};
> +\end{lstlisting}
> +
> +\item[VIRTIO_NET_CTRL_ROCE_GET_DMA_MR] Get the DMA Memory Region (MR).
> + associated with one protection domain.
I wonder what's the difference between VIRTIO_NET_CTRL_ROCE_GET_DMA_MR
and USR_MR. Can we unify them?
> + The command-specific-data is \field{virtio_rdma_cmd_get_dma_mr};
> + the ack-specific-data is \field{virtio_rdma_ack_get_dma_mr}.
> +
> +\begin{lstlisting}
> +enum virtio_ib_access_flags {
> + VIRTIO_IB_ACCESS_LOCAL_WRITE = (1 << 0),
Is LOCAL_READ implied to work always?
> + VIRTIO_IB_ACCESS_REMOTE_WRITE = (1 << 1),
> + VIRTIO_IB_ACCESS_REMOTE_READ = (1 << 2),
> +};
> +
> +struct virtio_rdma_cmd_get_dma_mr {
> + /* The handle of PD which the MR associated with */
> + le32 pdn;
> + /* MR's protection attributes, enum virtio_ib_access_flags */
> + le32 access_flags;
> +};
> +
> +struct virtio_rdma_ack_get_dma_mr {
> + /* The handle of MR */
> + le32 mrn;
> + /* MR's local access key */
> + le32 lkey;
> + /* MR's remote access key */
> + le32 rkey;
> +};
> +\end{lstlisting}
> +
> +\item[VIRTIO_NET_CTRL_ROCE_REG_USER_MR] Register a user Memory Region
> + associated with one Protection Domain.
> + The command-specific-data is \field{virtio_rdma_cmd_reg_user_mr};
> + the ack-specific-data is \field{virtio_rdma_ack_reg_user_mr}.
> +
> +\begin{lstlisting}
> +struct virtio_rdma_cmd_reg_user_mr {
> + /* The handle of PD which the MR associated with */
> + le32 pdn;
> + /* MR's protection attributes, enum virtio_ib_access_flags */
> + le32 access_flags;
> + /* Starting virtual address of MR */
> + le64 virt_addr;
I guess this is actually the I/O virtual address and the device is in
charge of translate it to the page arrays below?
> + /* Length of MR */
> + le64 length;
> + /* Size of the below page array */
> + le32 npages;
> + /* Padding */
> + le32 padding;
> + /* Array to store physical address of each page in MR */
> + le64 pages[];
How do device know the size of a page?
> +};
I believe this command can fail, we need to describe the error conditions.
> +
> +struct virtio_rdma_ack_reg_user_mr {
> + /* The handle of MR */
> + le32 mrn;
> + /* MR's local access key */
> + le32 lkey;
> + /* MR's remote access key */
> + le32 rkey;
> +};
> +\end{lstlisting}
> +
> +\item[VIRTIO_NET_CTRL_ROCE_DEREG_MR] De-register a Memory Region.
> + The command-specific-data is \field{virtio_rdma_cmd_dereg_mr};
> + no ack-specific-data.
> +
> +\begin{lstlisting}
> +struct virtio_rdma_cmd_dereg_mr {
> + /* The handle of MR */
> + le32 mrn;
> +};
> +\end{lstlisting}
> +
> +\item[VIRTIO_NET_CTRL_ROCE_CREATE_QP] Create a Queue Pair (Send Queue and Receive Queue).
> + The command-specific-data is \field{virtio_rdma_cmd_create_qp};
> + the ack-specific-data is \field{virtio_rdma_ack_create_qp}.
> +
> +\begin{lstlisting}
> +struct virtio_rdma_qp_cap {
> + /* Maximum number of outstanding WRs in SQ */
> + le32 max_send_wr;
> + /* Maximum number of outstanding WRs in RQ */
> + le32 max_recv_wr;
> + /* Maximum number of s/g elements per WR in SQ */
> + le32 max_send_sge;
> + /* Maximum number of s/g elements per WR in RQ */
> + le32 max_recv_sge;
> + /* Maximum number of data (bytes) that can be posted inline to SQ */
> + le32 max_inline_data;
> + /* Padding */
> + le32 padding;
> +};
> +
> +struct virtio_rdma_cmd_create_qp {
> + /* The handle of PD which the QP associated with */
> + le32 pdn;
> +#define VIRTIO_IB_QPT_SMI 0
> +#define VIRTIO_IB_QPT_GSI 1
> +#define VIRTIO_IB_QPT_RC 2
> +#define VIRTIO_IB_QPT_UC 3
> +#define VIRTIO_IB_QPT_UD 4
> + /* QP's type */
> + u8 qp_type;
> + /* If set, each WR submitted to the SQ generates a completion entry */
> + u8 sq_sig_all;
> + /* Padding */
> + u8 padding[2];
> + /* The index of CQ which the SQ associated with */
> + le32 send_cqn;
> + /* The index of CQ which the RQ associated with */
> + le32 recv_cqn;
> + /* QP's capabilities */
> + struct virtio_rdma_qp_cap cap;
> + /* Reserved for future */
> + le32 reserved[4];
> +};
> +
> +struct virtio_rdma_ack_create_qp {
> + /* The index of QP */
> + le32 qpn;
> +};
> +\end{lstlisting}
> +
> +\item[VIRTIO_NET_CTRL_ROCE_MODIFY_QP] Modify the attributes of a Queue Pair.
> + The command-specific-data is \field{virtio_rdma_cmd_modify_qp};
> + no ack-specific-data.
> +
> +\begin{lstlisting}
> +struct virtio_rdma_global_route {
> + /* Destination GID or MGID */
> + u8 dgid[16];
> + /* Flow label */
> + le32 flow_label;
> + /* Source GID index */
> + u8 sgid_index;
> + /* Hop limit */
> + u8 hop_limit;
> + /* Traffic class */
> + u8 traffic_class;
> + /* Padding */
> + u8 padding;
> +};
> +
> +struct virtio_rdma_ah_attr {
> + /* Global Routing Header (GRH) attributes */
> + virtio_rdma_global_route grh;
> + /* Destination MAC address */
> + u8 dmac[6];
> + /* Reserved for future */
> + u8 reserved[10];
> +};
> +
> +enum virtio_ib_qp_attr_mask {
> + VIRTIO_IB_QP_STATE = (1 << 0),
> + VIRTIO_IB_QP_CUR_STATE = (1 << 1),
> + VIRTIO_IB_QP_ACCESS_FLAGS = (1 << 2),
> + VIRTIO_IB_QP_QKEY = (1 << 3),
> + VIRTIO_IB_QP_AV = (1 << 4),
> + VIRTIO_IB_QP_PATH_MTU = (1 << 5),
> + VIRTIO_IB_QP_TIMEOUT = (1 << 6),
> + VIRTIO_IB_QP_RETRY_CNT = (1 << 7),
> + VIRTIO_IB_QP_RNR_RETRY = (1 << 8),
> + VIRTIO_IB_QP_RQ_PSN = (1 << 9),
> + VIRTIO_IB_QP_MAX_QP_RD_ATOMIC = (1 << 10),
> + VIRTIO_IB_QP_MIN_RNR_TIMER = (1 << 11),
> + VIRTIO_IB_QP_SQ_PSN = (1 << 12),
> + VIRTIO_IB_QP_MAX_DEST_RD_ATOMIC = (1 << 13),
> + VIRTIO_IB_QP_CAP = (1 << 14),
> + VIRTIO_IB_QP_DEST_QPN = (1 << 15),
> + VIRTIO_IB_QP_RATE_LIMIT = (1 << 16),
> +};
Do we need to explain the above error codes? Or it's simply a map from IB spec?
> +
> +enum virtio_ib_qp_state {
> + VIRTIO_IB_QPS_RESET,
> + VIRTIO_IB_QPS_INIT,
> + VIRTIO_IB_QPS_RTR,
> + VIRTIO_IB_QPS_RTS,
> + VIRTIO_IB_QPS_SQD,
> + VIRTIO_IB_QPS_SQE,
> + VIRTIO_IB_QPS_ERR
> +};
> +
> +enum virtio_ib_mtu {
> + VIRTIO_IB_MTU_256 = 1,
> + VIRTIO_IB_MTU_512 = 2,
> + VIRTIO_IB_MTU_1024 = 3,
> + VIRTIO_IB_MTU_2048 = 4,
> + VIRTIO_IB_MTU_4096 = 5
> +};
> +
> +struct virtio_rdma_cmd_modify_qp {
> + /* The index of QP */
> + le32 qpn;
> + /* The mask of attributes needs to be modified, enum virtio_ib_qp_attr_mask */
> + le32 attr_mask;
> + /* Move the QP to this state, enum virtio_ib_qp_state */
> + u8 qp_state;
> + /* Current QP state, enum virtio_ib_qp_state */
> + u8 cur_qp_state;
> + /* Path MTU (valid only for RC/UC QPs), enum virtio_ib_mtu */
> + u8 path_mtu;
> + /* Number of outstanding RDMA Read operations on destination QP (valid only for RC QPs) */
> + u8 max_rd_atomic;
> + /* Number of responder resources for handling incoming RDMA Read operations (valid only for RC QPs) */
> + u8 max_dest_rd_atomic;
> + /* Minimum RNR (Receiver Not Ready) NAK timer (valid only for RC QPs) */
> + u8 min_rnr_timer;
> + /* Local ack timeout (valid only for RC QPs) */
> + u8 timeout;
> + /* Retry count (valid only for RC QPs) */
> + u8 retry_cnt;
> + /* RNR retry (valid only for RC QPs) */
> + u8 rnr_retry;
> + /* Padding */
> + u8 padding[7];
> + /* Q_Key for the QP (valid only for UD QPs) */
> + le32 qkey;
> + /* PSN for RQ (valid only for RC/UC QPs) */
> + le32 rq_psn;
> + /* PSN for SQ */
> + le32 sq_psn;
> + /* Destination QP number (valid only for RC/UC QPs) */
> + le32 dest_qp_num;
> + /* Mask of enabled remote access operations (valid only for RC/UC QPs), enum virtio_ib_access_flags */
> + le32 qp_access_flags;
> + /* Rate limit in kbps for packet pacing */
> + le32 rate_limit;
> + /* QP capabilities */
> + struct virtio_rdma_qp_cap cap;
> + /* Address Vector (valid only for RC/UC QPs) */
> + struct virtio_rdma_ah_attr ah_attr;
> + /* Reserved for future */
> + le32 reserved[4];
> +};
> +\end{lstlisting}
> +
> +\item[VIRTIO_NET_CTRL_ROCE_QUERY_QP] Query the attributes of a Queue Pair.
> + The command-specific-data is \field{virtio_rdma_cmd_query_qp};
> + the ack-specific-data is \field{virtio_rdma_ack_query_qp}.
> +
> +\begin{lstlisting}
> +struct virtio_rdma_cmd_query_qp {
> + /* The index of QP */
> + le32 qpn;
> + /* The mask of attributes need to be queried, enum virtio_ib_qp_attr_mask */
> + le32 attr_mask;
> +};
> +
> +struct virtio_rdma_ack_query_qp {
Any chance to unify this with virtio_rdma_cmd_modify_qp?
> + /* Move the QP to this state, enum virtio_ib_qp_state */
> + u8 qp_state;
> + /* Path MTU (valid only for RC/UC QPs), enum virtio_ib_mtu */
> + u8 path_mtu;
> + /* Is the SQ draining */
> + u8 sq_draining;
> + /* Number of outstanding RDMA read operations on destination QP (valid only for RC QPs) */
> + u8 max_rd_atomic;
> + /* Number of responder resources for handling incoming RDMA read operations (valid only for RC QPs) */
> + u8 max_dest_rd_atomic;
> + /* Minimum RNR NAK timer (valid only for RC QPs) */
> + u8 min_rnr_timer;
> + /* Local ack timeout (valid only for RC QPs) */
> + u8 timeout;
> + /* Retry count (valid only for RC QPs) */
> + u8 retry_cnt;
> + /* RNR retry (valid only for RC QPs) */
> + u8 rnr_retry;
> + /* Padding */
> + u8 padding[7];
> + /* Q_Key for the QP (valid only for UD QPs) */
> + le32 qkey;
> + /* PSN for RQ (valid only for RC/UC QPs) */
> + le32 rq_psn;
> + /* PSN for SQ */
> + le32 sq_psn;
> + /* Destination QP number (valid only for RC/UC QPs) */
> + le32 dest_qp_num;
> + /* Mask of enabled remote access operations (valid only for RC/UC QPs), enum virtio_ib_access_flags */
> + le32 qp_access_flags;
> + /* Rate limit in kbps for packet pacing */
> + le32 rate_limit;
> + /* QP capabilities */
> + struct virtio_rdma_qp_cap cap;
> + /* Address Vector (valid only for RC/UC QPs) */
> + struct virtio_rdma_ah_attr ah_attr;
> + /* Reserved for future */
> + le32 reserved[4];
> +};
> +\end{lstlisting}
> +
> +\item[VIRTIO_NET_CTRL_ROCE_DESTROY_QP] Destroy a Queue Pair.
> + The command-specific-data is \field{virtio_rdma_cmd_destroy_qp};
> + no ack-specific-data.
What happen to the pending requests? Will the device wait for the
completion or not?
> +
> +\begin{lstlisting}
> +struct virtio_rdma_cmd_destroy_qp {
> + /* The index of QP */
> + le32 qpn;
> +};
> +\end{lstlisting}
> +
> +\item[VIRTIO_NET_CTRL_ROCE_CREATE_AH] Create a Address Handle (AH).
> + The command-specific-data is \field{virtio_rdma_cmd_create_ah};
> + the ack-specific-data is \field{virtio_rdma_ack_create_ah}.
> +
> +\begin{lstlisting}
> +struct virtio_rdma_cmd_create_ah {
> + /* The handle of PD which the AH associated with */
> + le32 pdn;
> + /* Padding */
> + le32 padding;
> + /* Address Vector */
> + struct virtio_rdma_ah_attr ah_attr;
> +};
> +
> +struct virtio_rdma_ack_create_ah {
> + /* The address handle */
> + le32 ah;
> +};
> +\end{lstlisting}
> +
> +\item[VIRTIO_NET_CTRL_ROCE_DESTROY_AH] Destroy a Address Handle.
> + The command-specific-data is \field{virtio_rdma_cmd_destroy_ah};
> + no ack-specific-data.
> +
> +\begin{lstlisting}
> +struct virtio_rdma_cmd_destroy_ah {
> + /* The handle of PD which the AH associated with */
> + le32 pdn;
> + /* The address handle */
> + le32 ah;
> +};
> +\end{lstlisting}
> +
> +\item[VIRTIO_NET_CTRL_ROCE_ADD_GID] Add a Global Identifier (GID).
> + The command-specific-data is \field{virtio_rdma_cmd_add_gid};
> + no ack-specific-data.
> +
> +\begin{lstlisting}
> +struct virtio_rdma_cmd_add_gid {
> + /* Index of GID */
> + le16 index;
> + /* Padding */
> + le16 padding[3];
> + /* GID to be added */
> + u8 gid[16];
> +};
> +\end{lstlisting}
> +
> +\item[VIRTIO_NET_CTRL_ROCE_DEL_GID] Delete a Global Identifier.
> + The command-specific-data is \field{virtio_rdma_cmd_del_gid};
> + no ack-specific-data.
> +
> +\begin{lstlisting}
> +struct virtio_rdma_cmd_del_gid {
> + /* Index of GID */
> + le16 index;
> +};
> +\end{lstlisting}
> +
> +\item[VIRTIO_NET_CTRL_ROCE_REQ_NOTIFY_CQ] Request a completion notification
> + on a Completion Queue.
> + The command-specific-data is \field{virtio_rdma_cmd_req_notify};
> + no ack-specific-data.
> +
> +\begin{lstlisting}
> +struct virtio_rdma_cmd_req_notify {
> + /* The index of CQ */
> + le32 cqn;
> +#define VIRTIO_IB_NOTIFY_SOLICITED (1 << 0)
> +#define VIRTIO_IB_NOTIFY_NEXT_COMPLETION (1 << 1)
Need to describe the differences on those two flags.
> + /* Notify flags */
> + le32 flags;
> +};
> +\end{lstlisting}
> +
> +\end{description}
> +
> +\drivernormative{\subparagraph}{RoCE Configuration}{Device Types / Network Device / Device Operation / Control Virtqueue / RoCE Configuration}
> +
> +A driver MUST initialize the completion virtqueue and fill it with
> +enough entries after command VIRTIO_NET_CTRL_ROCE_CREATE_CQ is
> +successfully executed.
> +
> +A driver MUST reset the completion virtqueue after
How to do the reset? Do you mean driver need to reset the indices?
> +command VIRTIO_NET_CTRL_ROCE_DESTROY_CQ is successfully executed.
> +
> +A driver MUST initialize the send virtqueue and receive virtqueue after
> +command VIRTIO_NET_CTRL_ROCE_CREATE_QP is successfully executed.
> +
> +A driver MUST reset the send virtqueue and receive virtqueue after
> +command VIRTIO_NET_CTRL_ROCE_DESTROY_QP is successfully executed.
>
> \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device
> Types / Network Device / Legacy Interface: Framing Requirements}
> @@ -4496,6 +5063,289 @@ \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device
> See \ref{sec:Basic
> Facilities of a Virtio Device / Virtqueues / Message Framing}.
>
> +\subsubsection{RoCE Support}\label{sec:Device Types / Network Device / Device Operation / RoCE Support}
> +
> +RDMA over Converged Ethernet (RoCE) is a network protocol that allows
> +remote direct memory access (RDMA) over an Ethernet network. To support
> +RoCE (if VIRTIO_NET_F_ROCE is negotiated), in addtion to the control
> +virtqueue support mentioned in \ref{sec:Device Types / Network Device /
> +Device Operation / Control Virtqueue / RoCE Configuration}, multiple
> +types of virtqueues including send virtqueue, receive virtqueue and
> +completion virtqueue are introduced.
> +
> +The send virtqueue contains elements that describe the data to be
> +transmitted.
> +
> +Requests (device-readable) have the following format:
> +
> +\begin{lstlisting}
> +enum virtio_ib_wr_opcode {
> + VIRTIO_IB_WR_RDMA_WRITE,
> + VIRTIO_IB_WR_RDMA_WRITE_WITH_IMM,
> + VIRTIO_IB_WR_SEND,
> + VIRTIO_IB_WR_SEND_WITH_IMM,
> + VIRTIO_IB_WR_RDMA_READ,
> +};
> +
> +struct virtio_rdma_sge {
> + le64 addr;
> + le32 length;
> + le32 lkey;
> +};
> +
> +struct virtio_rdma_sq_req {
> + /* User defined WR ID */
> + le64 wr_id;
> + /* WR opcode, enum virtio_ib_wr_opcode */
> + u8 opcode;
> +#define VIRTIO_IB_SEND_FENCE (1 << 0)
> +#define VIRTIO_IB_SEND_SIGNALED (1 << 1)
> +#define VIRTIO_IB_SEND_SOLICITED (1 << 2)
> +#define VIRTIO_IB_SEND_INLINE (1 << 3)
> + /* Flags of the WR properties */
> + u8 send_flags;
> + /* Padding */
> + le16 padding;
> + /* Immediate data (in network byte order) to send */
> + le32 imm_data;
> + union {
> + struct {
> + /* Start address of remote memory buffer */
> + le64 remote_addr;
> + /* Key of the remote MR */
> + le32 rkey;
> + } rdma;
> + struct {
> + /* Index of the destination QP */
> + le32 remote_qpn;
> + /* Q_Key of the destination QP */
> + le32 remote_qkey;
> + /* Address Handle */
> + le32 ah;
> + } ud;
> + /* Reserved for future */
> + le64 reserved[4];
> + };
> + /* Inline data */
> + u8 inline_data[512];
> + union {
> + /* Length of sg_list */
> + le32 num_sge;
> + /* Length of inline data */
> + le16 inline_len;
> + };
> + /* Reserved for future */
> + le32 reserved2[3];
> + /* Scatter/gather list */
> + struct virtio_rdma_sge sg_list[];
> +};
> +\end{lstlisting}
> +
> +The receive virtqueue contains elements that describe where to place incoming data.
> +
> +Requests (device-readable) have the following format:
> +
> +\begin{lstlisting}
> +struct virtio_rdma_rq_req {
> + /* User defined WR ID */
> + le64 wr_id;
> + /* Length of sg_list */
> + le32 num_sge;
> + /* Reserved for future */
> + le32 reserved[3];
> + /* Scatter/gather list */
> + struct virtio_rdma_sge sg_list[];
> +};
> +\end{lstlisting}
> +
> +The completion virtqueue is used to notify the completion of requests in
> +send virtqueue or receive virtqueue.
> +
> +Requests (device-writable) have the following format:
> +
> +\begin{lstlisting}
> +enum virtio_ib_wc_opcode {
> + VIRTIO_IB_WC_SEND,
> + VIRTIO_IB_WC_RDMA_WRITE,
> + VIRTIO_IB_WC_RDMA_READ,
> + VIRTIO_IB_WC_RECV,
> + VIRTIO_IB_WC_RECV_RDMA_WITH_IMM,
> +};
> +
> +enum virtio_ib_wc_status {
> + /* Operation completed successfully */
> + VIRTIO_IB_WC_SUCCESS,
> + /* Local Length Error */
> + VIRTIO_IB_WC_LOC_LEN_ERR,
> + /* Local QP Operation Error */
> + VIRTIO_IB_WC_LOC_QP_OP_ERR,
> + /* Local Protection Error */
> + VIRTIO_IB_WC_LOC_PROT_ERR,
> + /* Work Request Flushed Error */
> + VIRTIO_IB_WC_WR_FLUSH_ERR,
> + /* Bad Response Error */
> + VIRTIO_IB_WC_BAD_RESP_ERR,
> + /* Local Access Error */
> + VIRTIO_IB_WC_LOC_ACCESS_ERR,
> + /* Remote Invalid Request Error */
> + VIRTIO_IB_WC_REM_INV_REQ_ERR,
> + /* Remote Access Error */
> + VIRTIO_IB_WC_REM_ACCESS_ERR,
> + /* Remote Operation Error */
> + VIRTIO_IB_WC_REM_OP_ERR,
> + /* Transport Retry Counter Exceeded */
> + VIRTIO_IB_WC_RETRY_EXC_ERR,
> + /* RNR Retry Counter Exceeded */
> + VIRTIO_IB_WC_RNR_RETRY_EXC_ERR,
> + /* Remote Aborted Error */
> + VIRTIO_IB_WC_REM_ABORT_ERR,
> + /* Fatal Error */
> + VIRTIO_IB_WC_FATAL_ERR,
> + /* Response Timeout Error */
> + VIRTIO_IB_WC_RESP_TIMEOUT_ERR,
> + /* General Error */
> + VIRTIO_IB_WC_GENERAL_ERR
> +};
> +
> +struct virtio_rdma_cq_req {
> + /* User defined WR ID */
> + le64 wr_id;
> + /* Work completion status, enum virtio_ib_wc_status */
> + u8 status;
> + /* WR opcode, enum virtio_ib_wc_opcode */
> + u8 opcode;
> + /* Padding */
> + le16 padding;
> + /* Vendor error */
> + le32 vendor_err;
> + /* Number of bytes transferred */
> + le32 byte_len;
> + /* Immediate data (in network byte order) to send */
> + le32 imm_data;
> + /* Local QP number of completed WR */
> + le32 qp_num;
> + /* Source QP number (remote QP number) of completed WR (valid only for UD QPs) */
> + le32 src_qp;
> +#define VIRTIO_IB_WC_GRH (1 << 0)
> +#define VIRTIO_IB_WC_WITH_IMM (1 << 1)
> + /* Work completion flag */
> + le32 wc_flags;
> + /* Reserved for future */
> + le32 reserved[3];
> +};
> +\end{lstlisting}
> +
> +\paragraph{Send Operation}\label{sec:Device Types / Network Device / Device Operation / RoCE Support / Send Operation}
> +
> +The send operation allows us to send data to a remote QPâs Receive Queue.
> +The receiver MUST have previously posted a receive buffer to receive the data.
"MUST" keyword must belong to the normative section.
> +
> +To do a send operation, a request with \field{opcode} set to
> +VIRTIO_IB_WR_SEND or VIRTIO_IB_WR_SEND_WITH_IMM MUST be posted to the Send
> +Queue as one output descriptor and the device is notified of the new entry.
> +
> +\drivernormative{\subparagraph}{Send Operation}{Device Types / Network Device / Device Operation / RoCE Support / Send Operation}
> +
> +If VIRTIO_IB_SEND_INLINE is set in \field{send_flags}, the driver MUST fill
> +send buffer into \field{inline_data} field and set \field{inline_len} to the
> +length of the buffer. Otherwise, the driver MUST fill \field{sg_list} to
> +describe the buffer.
> +
> +\devicenormative{\subparagraph}{Send Operation}{Device Types / Network Device / Device Operation / RoCE Support / Send Operation}
> +
> +If \field{opcode} is not set to VIRTIO_IB_WR_SEND_WITH_IMM, the device MUST
> +ignore \field{imm_data}.
> +
> +If the QP type is UD, the device MUST validate \field{ud.ah}.
> +
> +If VIRTIO_IB_SEND_INLINE is not set in \field{send_flags}, the device MUST
> +validate the \field{addr}, \field{length} and \field{lkey} in \field{sg_list}.
> +
> +\paragraph{Receive Operation}\label{sec:Device Types / Network Device / Device Operation / RoCE Support / Receive Operation}
> +
> +The receive operation allows us to receive data from remote QP.
> +It's the corresponding operation to a send operation.
> +
> +To do a receive operation, a request MUST be posted to the Receive
> +Queue as one output descriptor and the device is notified of the new entry.
> +
I think we probably need to be more verbose as what has been done for
virtio-net.
That is, describe what need to be filled in virtio_rdma_rq_req in
details. (And do this for other operation as well)
> +\drivernormative{\subparagraph}{Receive Operation}{Device Types / Network Device / Device Operation / RoCE Support / Receive Operation}
> +
> +The driver MUST fill \field{sg_list} to describe the receive buffer.
> +
> +\devicenormative{\subparagraph}{Receive Operation}{Device Types / Network Device / Device Operation / RoCE Support / Receive Operation}
> +
> +The device MUST validate the \field{addr}, \field{length} and \field{lkey}
> +in \field{sg_list}.
> +
> +\paragraph{Write Operation}\label{sec:Device Types / Network Device / Device Operation / RoCE Support / Write Operation}
> +
> +The write operation allows us to write data to the local memory buffer
> +in remote side with no notification. The remote side wouldn't be aware
> +that this operation being done.
> +
> +To do a write operation, a request with \field{opcode} set to
> +VIRTIO_IB_WR_RDMA_WRITE or VIRTIO_IB_WR_RDMA_WRITE_WITH_IMM MUST be
> +posted to the Send Queue as one output descriptor and the device is
> +notified of the new entry.
> +
> +\drivernormative{\subparagraph}{Write Operation}{Device Types / Network Device / Device Operation / RoCE Support / Write Operation}
> +
> +The driver MUST fill \field{sg_list} to describe the write buffer.
So sg is a must even if the driver want to use imm?
> +
> +The driver MUST fill \field{rdma.remote_addr} and \field{rdma.rkey} to
> +identify the remote buffer.
> +
> +\devicenormative{\subparagraph}{Write Operation}{Device Types / Network Device / Device Operation / RoCE Support / Write Operation}
> +
> +If \field{opcode} is not set to VIRTIO_IB_WR_RDMA_WRITE_WITH_IMM, the device
> +MUST ignore \field{imm_data}.
> +
> +The device MUST validate the \field{addr}, \field{length} and \field{lkey}
> +in \field{sg_list}.
> +
> +\paragraph{Read Operation}\label{sec:Device Types / Network Device / Device Operation / RoCE Support / Read Operation}
> +
> +The read operation allows us to read data from the local memory buffer
> +in remote side with no notification. The remote side wouldn't be aware
> +that this operation being done.
> +
> +To do a read operation, a request with \field{opcode} set to
> +VIRTIO_IB_WR_RDMA_READ MUST be posted to the Send Queue as one output
> +descriptor and the device is notified of the new entry.
> +
> +\drivernormative{\subparagraph}{Read Operation}{Device Types / Network Device / Device Operation / RoCE Support / Read Operation}
> +
> +The driver MUST fill \field{sg_list} to describe the read buffer.
> +
> +The driver MUST fill \field{rdma.remote_addr} and \field{rdma.rkey} to
> +identify the remote buffer.
> +
> +\devicenormative{\subparagraph}{Read Operation}{Device Types / Network Device / Device Operation / RoCE Support / Read Operation}
> +
> +The device MUST validate the \field{addr}, \field{length} and \field{lkey}
> +in \field{sg_list}.
> +
> +\paragraph{Completion Notification}\label{sec:Device Types / Network Device / Device Operation / RoCE Support / Completion Notification}
> +
> +After above operation is completed, a completion notification MUST
> +be triggered by the device.
For "completion notification", do you mean the virtqueue notification
of cq or the making the buffer than contains cqe used?
> To achieve that, the device MUST consume
> +an entry of the Completion Queue associated with the Send Queue/Receive
> +Queue which the operation belongs to.
> +
> +\drivernormative{\subparagraph}{Completion Notification}{Device Types / Network Device / Device Operation / RoCE Support / Completion Notification}
> +
> +The driver MUST fill the Completion Queue with enough entries previously.
What do you mean by "previously"? What happens if there's no sufficient cqe?
Thanks
> +
> +\devicenormative{\subparagraph}{Completion Notification}{Device Types / Network Device / Device Operation / RoCE Support / Completion Notification}
> +
> +If \field{imm_data} is valid, the device MUST set VIRTIO_IB_WC_WITH_IMM to
> +\field{wc_flags}.
> +
> +The device MUST set \field{wr_id} to the value of \field{wr_id} of
> +corresponding \field{struct virtio_rdma_sq_req} or
> +\field{struct virtio_rdma_rq_req}.
> +
> \section{Block Device}\label{sec:Device Types / Block Device}
>
> The virtio block device is a simple virtual block device (ie.
> --
> 2.11.0
>
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]