[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]
Subject: Re: [PATCH v12] virtio-net: support device stats
å 2022/3/15 äå11:24, Xuan Zhuo åé:
This patch allows the driver to obtain some statistics from the device. In the back-end implementation, we can count a lot of such information, which can be used for debugging and judging the running status of the back-end. We hope to directly display it to the user through ethtool. To get stats atomically, try to get stats for all queue pairs in one command. Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com> Suggested-by: Michael S. Tsirkin <mst@redhat.com> --- conformance.tex | 2 + content.tex | 406 +++++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 405 insertions(+), 3 deletions(-) diff --git a/conformance.tex b/conformance.tex index 42f8537..c67f877 100644 --- a/conformance.tex +++ b/conformance.tex @@ -142,6 +142,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode} \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Offloads State Configuration / Setting Offloads State} \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) } +\item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Device Stats} \end{itemize}\conformance{\subsection}{Block Driver Conformance}\label{sec:Conformance / Driver Conformance / Block Driver Conformance}@@ -401,6 +402,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Gratuitous Packet Sending} \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode} \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) / RSS processing} +\item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Device Stats} \end{itemize}\conformance{\subsection}{Block Device Conformance}\label{sec:Conformance / Device Conformance / Block Device Conformance}diff --git a/content.tex b/content.tex index c6f116c..81f325d 100644 --- a/content.tex +++ b/content.tex @@ -3092,6 +3092,9 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control channel.+\item[VIRTIO_NET_F_CTRL_STATS(55)] Device can provide device-level statistics+ to the driver through the control channel. + \item[VIRTIO_NET_F_HOST_USO (56)] Device can receive USO packets. Unlike UFO (fragmenting the packet) the USO splits large UDP packet to several segments when each of these smaller packets has UDP header. @@ -3137,6 +3140,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device \item[VIRTIO_NET_F_GUEST_ANNOUNCE] Requires VIRTIO_NET_F_CTRL_VQ. \item[VIRTIO_NET_F_MQ] Requires VIRTIO_NET_F_CTRL_VQ. \item[VIRTIO_NET_F_CTRL_MAC_ADDR] Requires VIRTIO_NET_F_CTRL_VQ. +\item[VIRTIO_NET_F_CTRL_STATS] Requires VIRTIO_NET_F_CTRL_VQ. \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6. \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ. \end{description} @@ -4015,6 +4019,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi u8 command; u8 command-specific-data[]; u8 ack; + u8 command-specific-data-reply[]; };/* ack values */@@ -4023,9 +4028,11 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \end{lstlisting}The \field{class}, \field{command} and command-specific-data are set by the-driver, and the device sets the \field{ack} byte. There is little it can -do except issue a diagnostic if \field{ack} is not -VIRTIO_NET_OK. +driver, and the device sets the \field{ack} byte and optionally +\field{command-specific-data-reply}. There is little the driver can +do except issue a diagnostic if \field{ack} is not VIRTIO_NET_OK. + +The command VIRTIO_NET_CTRL_STATS_GET contains \field{command-specific-data-reply}.\paragraph{Packet Receive Filtering}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Packet Receive Filtering}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Setting Promiscuous Mode}%old label for latexdiff @@ -4471,6 +4478,399 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi according to the native endian of the guest rather than (necessarily when not using the legacy interface) little-endian.+\paragraph{Device Stats}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Device Stats}+ +If the VIRTIO_NET_F_CTRL_STATS feature is negotiated, the driver can +get the device stats from the device in \field{command-specific-data-reply}. + +To get the stats, the following definitions are used: +\begin{lstlisting} +#define VIRTIO_NET_CTRL_STATS 6 +#define VIRTIO_NET_CTRL_STATS_GET 0 + +#define VIRTIO_NET_STATS_TYPE_CVQ 0 +#define VIRTIO_NET_STATS_TYPE_RX_BASIC 1 +#define VIRTIO_NET_STATS_TYPE_RX_CSUM 2 +#define VIRTIO_NET_STATS_TYPE_RX_GSO 3 +#define VIRTIO_NET_STATS_TYPE_RX_RESET 4 +#define VIRTIO_NET_STATS_TYPE_TX_BASIC 5 +#define VIRTIO_NET_STATS_TYPE_TX_CSUM 6 +#define VIRTIO_NET_STATS_TYPE_TX_GSO 7 +#define VIRTIO_NET_STATS_TYPE_TX_RESET 8 + +\end{lstlisting} + +Use the command VIRTIO_NET_CTRL_STATS_GET and \field{command-specific-data} +containing struct virtio_net_ctrl_queue_stats to get the device stats. +The result is returned by \field{command-specific-data-reply}. +The stats ware returned in the order of the type specified in the
s/ware/were/ and s/type/types/ ?
+\field{virtio_net_ctrl_queue_stats}. + +The following layout structures are used: + +\field{command-specific-data} +\begin{lstlisting} +struct virtio_net_ctrl_queue_stats { + u16 nstats; + struct { + u16 queue_num; + u16 type; + } stats[]; +}; +\end{lstlisting} + +\field{command-specific-data-reply} +\begin{lstlisting} +struct virtio_net_stats_cvq { + le64 command_num; + le64 ok_num; +}; + +struct virtio_net_stats_rx_basic { + le64 rx_packets; + le64 rx_bytes; + + le64 rx_notification; + le64 rx_interrupt; + + le64 rx_drop; + le64 rx_drop_overruns; +}; + +struct virtio_net_stats_rx_csum { + le64 rx_csum_valid; + le64 rx_needs_csum; + le64 rx_csum_bad; + le64 rx_csum_none; +}; + +struct virtio_net_stats_rx_gso { + le64 rx_gso_packets; + le64 rx_gso_bytes; + le64 rx_gso_packets_coalesced; + le64 rx_gso_bytes_coalesced; + le64 rx_gso_segments; + le64 rx_gso_segments_bytes; +}; + +struct virtio_net_stats_rx_reset { + le64 rx_reset; +}; + +struct virtio_net_stats_tx_basic { + le64 tx_packets; + le64 tx_bytes; + + le64 tx_notification; + le64 tx_interrupt; + + le64 tx_drop; + le64 tx_drop_malformed; +}; + +struct virtio_net_stats_tx_csum { + le64 tx_csum_none; + le64 tx_needs_csum; +}; + +struct virtio_net_stats_tx_gso { + le64 tx_gso_packets; + le64 tx_gso_bytes; + le64 tx_gso_packets_split; + le64 tx_gso_bytes_split; + le64 tx_gso_segments; + le64 tx_gso_segments_bytes; +}; + +struct virtio_net_stats_tx_reset { + le64 tx_reset; +}; + +\end{lstlisting} + +\begin{description} + \item [nstats] + The number of \field{stats}.
This looks not necessary since it can be deduced from the buffer length.
+ + \item [queue_num] + The number of the virtqueue to obtain the statistics. + + \item [type] + The type of the stats to be obtained. + +\end{description} + +Correspondence between the vq type, the stats type, the stats structure and the +required features. +\begin{tabular}{ |l|l|l|l| } + \hline + VQ Type & Stats Type & Stats Structure & Features \\ \hline + + controlq & VIRTIO_NET_STATS_TYPE_CVQ & virtio_net_stats_cvq & \\ \hline + + \multirow{4}*{receiveq} & VIRTIO_NET_STATS_TYPE_RX_BASIC & virtio_net_stats_rx_basic & \\ \cline{2-4} + & VIRTIO_NET_STATS_TYPE_RX_CSUM & virtio_net_stats_rx_csum & VIRTIO_NET_F_GUEST_CSUM \\ \cline{2-4} + & VIRTIO_NET_STATS_TYPE_RX_GSO & virtio_net_stats_rx_gso & VIRTIO_NET_F_GUEST_TSO4 or\newline + VIRTIO_NET_F_GUEST_TSO6 or\newline + VIRTIO_NET_F_GUEST_UFO \\ \cline{2-4} + & VIRTIO_NET_STATS_TYPE_RX_RESET & virtio_net_stats_rx_reset & VIRTIO_F_RING_RESET \\ \hline + + \multirow{4}*{transmitq} & VIRTIO_NET_STATS_TYPE_TX_BASIC & virtio_net_stats_tx_basic & \\ \cline{2-4} + & VIRTIO_NET_STATS_TYPE_TX_CSUM & virtio_net_stats_tx_csum & VIRTIO_NET_F_CSUM \\ \cline{2-4} + & VIRTIO_NET_STATS_TYPE_TX_GSO & virtio_net_stats_tx_gso & VIRTIO_NET_F_HOST_TSO4 or\newline + VIRTIO_NET_F_HOST_TSO6 or\newline + VIRTIO_NET_F_HOST_USO or\newline + VIRTIO_NET_F_HOST_UFO \\ \cline{2-4} + & VIRTIO_NET_STATS_TYPE_TX_RESET & virtio_net_stats_tx_reset & VIRTIO_F_RING_RESET \\ + \hline +\end{tabular} + + +\subparagraph{Controlq Stats}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Device Stats / Controlq Stats} + +The structure corresponding to the controlq stats is virtio_net_stats_cvq. + +\begin{description} + \item [command_num] + The number of commands, including the current command. + + \item [ok_num] + The number of commands (including the current command) where the ack was VIRTIO_NET_OK. +\end{description} + + +\subparagraph{Receiveq Basic Stats}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Device Stats / Receiveq Basic Stats} + +The structure corresponding to the receiveq basic stats is virtio_net_stats_rx_basic. + +Receiveq basic stats doesn't need any features, as long as the device supports +VIRTIO_NET_F_CTRL_STATS. The following are the receiveq basic stats. + +\begin{description} + \item [rx_packets] + The number of packets received by device (not the packets passed to the + guest), including the dropped packets by device. + + \item [rx_bytes] + The number of bytes received by device (not the packets passed to the + guest), including the dropped packets by device. + + \item [rx_notification] + The number of driver notifications. + + \item [rx_interrupt] + The number of device interrupts. + + \item [rx_drop] + The number of packets dropped by the receiveq. Contains all kinds of + packet drop. + + \item [rx_drop_overruns] + The number of packets dropped by the receiveq when no more descriptors + were available. + +\end{description} + +\subparagraph{Transmitq Basic Stats}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Device Stats / Transmitq Basic Stats} + +The structure corresponding to the transmitq basic stats is virtio_net_stats_tx_basic. + +Transmitq basic stats doesn't need any features, as long as the device supports +VIRTIO_NET_F_CTRL_STATS. The following are the transmitq basic stats. + +\begin{description} + \item [tx_packets] + The number of packets sent by device (not the packets got from the + guest), excluding the dropped packets by device.
"packets dropped by device"?
+ + \item [tx_bytes] + The number of bytes sent by device (not the packets got from the + guest), excluding the dropped packets by device. + + \item [tx_notification] + The number of driver notifications. + + \item [tx_interrupt] + The number of device interrupts. + + \item [tx_drop] + The number of packets dropped by the transmitq. Contains all kinds of + packet drop. + + \item [tx_drop_malformed] + The number of packets dropped when the descriptor is in an error state. + For example, the buffer is too short.
I wonder if "tx_erros" is better (this is what I see from at least two other NIC vendors).
+ +\end{description} + +\subparagraph{Receiveq CSUM Stats}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Device Stats / Receiveq CSUM Stats} + +The structure corresponding to the receiveq csum stats is virtio_net_stats_rx_csum. + +Only after the VIRTIO_NET_F_GUEST_CSUM negotiation is successful, the receiveq +csum stats can be obtained. + +The following are the receiveq csum stats: + +\begin{description} + \item [rx_csum_valid] + The number of packets with VIRTIO_NET_HDR_F_DATA_VALID.
A question, it looks to me the stats refer to the phy -> device counters not the device -> driver counter (which could be count by the driver).
If this is true, technically, device can't receive a packet with VIRTIO_NET_HDR_F_DATA_VALID.
FYI, e1000e had: ethtool -S enp0s31f6 | grep csum ÂÂÂÂ rx_csum_offload_good: 601959 ÂÂÂÂ rx_csum_offload_errors: 0
+ + \item [rx_needs_csum] + The number of packets with VIRTIO_NET_HDR_F_NEEDS_CSUM. + + \item [rx_csum_bad] + The number of packets with abnormal csum. + + \item [rx_csum_none] + The number of packets without hardware csum. The packet here refers to + the non-TCP/UDP packet that the backend cannot recognize.
This is probably not correct. We may have a guest without VIRTIO_NET_F_GUEST_CSUM support.
+ +\end{description} + +\subparagraph{Transmitq CSUM Stats}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Device Stats / Transmitq CSUM Stats} + +The structure corresponding to the transmitq csum stats is virtio_net_stats_tx_csum. + +Only after the VIRTIO_NET_F_CSUM negotiation is successful, the transmitq csum +stats can be obtained. + +The following are the transmitq csum stats: + +\begin{description} + \item [tx_csum_none] + The number of packets that didn't require hardware csum. + + \item [tx_needs_csum] + The number of packets that required hardware csum. + +\end{description} + +\subparagraph{Receiveq GSO Stats}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Device Stats / Receiveq GSO Stats} + +The structure corresponding to the receiveq gso stats is virtio_net_stats_rx_gso. + +If one of the VIRTIO_NET_F_GUEST_TSO4, TSO6, or UFO options have +been negotiated, the receiveq gso stats can be obtained.
We probably need to use "GSO" instead of "gso" for the whole patch.
+ +Rx gso packets refer to packets passed by the device to the driver where +\field{gso_type} is not VIRTIO_NET_HDR_GSO_NONE. + +\begin{description} + \item [rx_gso_packets] + The number of the rx gso packets. + + \item [rx_gso_bytes] + The number of bytes(excluding the virtio net header) of the rx gso packets. + + \item [rx_gso_packets_coalesced] + The number of the rx gso packages generated by coalescing.
For "packages", did you mean "packets" actually? Does it work only if RSC is negotiated?
+ + \item [rx_gso_bytes_coalesced] + The number of bytes(excluding the virtio net header) of the rx gso packets generated by coalescing. + + \item [rx_gso_segments] + The number of coalesced segments.
If we do VM2VM traffic, we can receive GSO packets directly without coalescing. Do we count this packet here?
+ + \item [rx_gso_segments_bytes] + The number of bytes of coalesced segments. + +\end{description} + +\subparagraph{Transmitq GSO Stats}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Device Stats / Transmitq GSO Stats} + +The structure corresponding to the transmitq gso stats is virtio_net_stats_tx_gso. + +If one of the VIRTIO_NET_F_HOST_TSO4, TSO6, USO or UFO options have +been negotiated, the transmitq gso stats can be obtained. + +Tx gso packets refer to packets passed by the driver to the device where +\field{gso_type} is not VIRTIO_NET_HDR_GSO_NONE. + +\begin{description} + \item [tx_gso_packets] + The number of the tx gso packets. + + \item [tx_gso_bytes] + The number of bytes(excluding the virtio net header) of the tx gso packets. + + \item [tx_gso_packets_split] + The number of the tx gso packets that been split. + + \item [tx_gso_bytes_split] + The number of bytes(excluding the virtio net header) of the tx gso packets that been split.
S/packet/bytes/? And does it include the L2/L3/L4 headers of the segmented packets?
+ + \item [tx_gso_segments] + The number of segments split from the gso package.
For package, I think you meant packet actually.
+ + \item [tx_gso_segments_bytes] + The number of bytes(excluding the virtio net header) of segments split from the gso package. +\end{description} + +\subparagraph{Receiveq Reset Stats}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Device Stats / Receiveq Reset Stats} + +The structure corresponding to the receiveq reset stats is virtio_net_stats_rx_reset. + +Only when VIRTIO_F_RING_RESET is successfully negotiated, the receiveq reset stats +can be obtained.
Not a native speaker, but I think we can remove "successfully" here.
+ +See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset} +for more about \field{rx_reset}. + +\begin{description} + \item [rx_reset] + The number of receiveq resets. +\end{description} + +\subparagraph{Transmitq Reset Stats}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Device Stats / Transmitq Reset Stats} + +The structure corresponding to the transmitq reset stats is virtio_net_stats_tx_reset. + +Only when VIRTIO_F_RING_RESET is successfully negotiated, the transmitq reset stats +can be obtained. + +See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset} +for more about \field{tx_reset}. + +\begin{description} + \item [tx_reset] + The number of transmitq resets. +\end{description} + +\devicenormative{\subparagraph}{Device Stats}{Device Types / Network Device / Device Operation / Control Virtqueue / Device Stats} + +If virtio_net_ctrl_queue_stats is incorrect (such as the following), the device +MUST set \field{ack} to VIRTIO_NET_ERR. Even if there is only one error, +the device MUST abort the entire command.
I guess "fail" is better than "abort".
+\begin{itemize} + \item \field{queue_num} exceeds the queue range. + \item \field{type} is not a known value. + \item The type of vq does not match \field{type}. E.g. the driver tries to query + RX stats through a TX index. + \item The feature corresponding to the specified \field{type} was not successfully + negotiated. + \item The size of the buffer allocated by the driver for \field{command-specific-data-reply} + is less than the total size of the stats specialed by + \field{virtio_net_ctrl_queue_stats}. +\end{itemize} + +The device MUST write the requested stats structures in +\field{command-specific-data-reply} in the order specified by the structure +virtio_net_ctrl_queue_stats.
Are the counters reset during device reset? Thanks
+ +\drivernormative{\subparagraph}{Device Stats}{Device Types / Network Device / Device Operation / Control Virtqueue / Device Stats} + +When a driver tries to obtain a certain stats, it MUST confirm that the relevant +feature negotiation is successful. + +\field{type} in struct virtio_net_ctrl_queue_stats MUST correspond to the vq +specified by \field{queue_num}. + +The \field{command-specific-data-reply} buffer allocated by the driver MUST be +able to hold all the stats specified by virtio_net_ctrl_queue_stats. + +When the driver reads the response, it MUST read +\field{command-specific-data-reply} one by one based on the \field{type}.\subsubsection{Legacy Interface: Framing Requirements}\label{sec:DeviceTypes / Network Device / Legacy Interface: Framing Requirements}
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]