Re: [virtio] [PATCH requirements v5 3/7] net-features: Add low latency receive queue requirements

[Stefan Hajnoczi <stefanha@redhat.com>]

On Fri, Aug 18, 2023 at 07:35:53AM +0300, Parav Pandit wrote:
> +### 3.2.2 Low latency rx virtqueue
> +0. Design goal:
> +   a. Keep packet metadata and buffer data together which is consumed by driver
> +      layer and make it available in a single cache line of cpu
> +   b. Instead of having per packet descriptors which is complex to scale for
> +      the device, supply the page directly to the device to consume it based
> +      on packet size
> +1. The device should be able to write a packet receive completion that consists
> +   of struct virtio_net_hdr (or similar) and a buffer id using a single DMA write
> +   PCIe TLP.
> +2. The device should be able to perform DMA writes of multiple packets
> +   completions in a single DMA transaction up to the PCIe maximum write limit
> +   in a transaction.
> +3. The device should be able to zero pad packet write completion to align it to
> +   64B or CPU cache line size whenever possible.
> +4. An example of the above DMA completion structure:
> +
> +```
> +/* Constant size receive packet completion */
> +struct vnet_rx_completion {
> +   u16 flags;
> +   u16 id; /* buffer id */
> +   u8 gso_type;
> +   u8 reserved[3];
> +   le16 gso_hdr_len;
> +   le16 gso_size;
> +   le16 csum_start;
> +   le16 csum_offset;
> +   u16 reserved2;
> +   u64 timestamp; /* explained later */
> +   u8 padding[];
> +};
> +```
> +5. The driver should be able to post constant-size buffer pages on a receive
> +   queue which can be consumed by the device for an incoming packet of any size
> +   from 64B to 9K bytes.
> +6. The device should be able to know the constant buffer size at receive
> +   virtqueue level instead of per buffer level.
> +7. The device should be able to indicate when a full page buffer is consumed,
> +   which can be recycled by the driver when the packets from the completed
> +   page is fully consumed.
> +8. The device should be able to consume multiple pages for a receive GSO stream.

If I understand correctly there is no longer a 1:1 correspondence
between driver-supplied rx pages (available buffers) and received
packets (used buffers). Instead, the device consumes portions of
driver-supplied rx pages as needed and notifies the driver, and the
entire rx page is marked used later when it has been fully consumed.

The virtqueue model is based on submitting available buffers and
completing used buffers, not individual DMA transfers. It's not possible
to do DMA piecewise in this model. If you think about a VIRTIO over TCP
transport that uses message-passing for available and used buffers, then
it's clear the rx page approach breaks the model because only entire
virtqueues buffers can be marked used (there is a 1:1 correspondence
between available buffers and used buffers).

Two options:
1. Extend the virtqueue model to support this.
2. Document this violation of the virtqueue model clearly but treat it
   as an exception that may lead to complications in the future (e.g.
   incompatibility with VIRTIO over TCP).

I think it's worth investigating #1 to see whether the virtqueue model
can be extended cleanly.

Stefan

Attachment: signature.asc
Description: PGP signature