Re: [virtio] [PATCH] virtqueue: flexible layout, size, alignment

[Rusty Russell <rusty@au1.ibm.com>]

"Michael S. Tsirkin" <mst@redhat.com> writes:
> Transports can now lay out available/used/descriptor
> regions in a flexible way.
> This is useful for embedded systems to save memory,
> and for large systems to reduce the need for
> physically-contigious memory.
>
> This does not add a way to actually program this
> in any of the transports, so it's not useful by
> itself, a follow-up patch with add a way to
> program this for PCI.
>
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  virtio-v1.0-wd01-part1-specification.txt | 52 ++++++++++++++++++++++++++++----
>  1 file changed, 46 insertions(+), 6 deletions(-)
>
> diff --git a/virtio-v1.0-wd01-part1-specification.txt b/virtio-v1.0-wd01-part1-specification.txt
> index fcd9fd7..9a40973 100644
> --- a/virtio-v1.0-wd01-part1-specification.txt
> +++ b/virtio-v1.0-wd01-part1-specification.txt
> @@ -223,6 +223,45 @@ transmit and one for receive.  Each queue has a 16-bit queue size
>  parameter, which sets the number of entries and implies the total size
>  of the queue.
>  
> +Each virtqueue consists of three parts:
> +
> +	Descriptor Table
> +	Available Ring
> +	Used Ring
> +
> +where each part is physically-contiguous in guest memory,
> +and has different alignment requirements.
> +
> +The Queue Size field controls the total number of bytes
> +required for each part of the virtqueue.
> +
> +The memory aligment and size requirements, in bytes, of each part of the
> +virtqueue are summarized in the following table (qsz is the Queue Size field):
> +
> ++------------+---------------------------------+
> +| Virtqueue Part    | Alignment | Size         |
> ++------------+---------------------------------+
> ++------------+---------------------------------+
> +| Descriptor Table  | 16        | 16 * qsz     |

Alignment 16: isn't that overkill?

> ++------------+---------------------------------+
> +| Available Ring    | 2         | 6 + 2 * qsz  |
> ++------------+---------------------------------+
> +| Used Ring         | 4         | 6 + 4 * qsz  |
> ++------------+---------------------------------+
> +
> +When the driver wants to send a buffer to the device, it fills in
> +a slot in the descriptor table (or chains several together), and
> +writes the descriptor index into the available ring.  It then
> +notifies the device. When the device has finished a buffer, it
> +writes the descriptor into the used ring, and sends an interrupt.
> +
> +
> +100.100.4.1. Legacy Interfaces: A Note on Virtqueue Layout
> +--------------------------------------
> +
> +For Legacy Interfaces, several additional
> +restrictions are placed on the virtqueue layout:
> +
>  Each virtqueue occupies two or more physically-contiguous pages
>  (usually defined as 4096 bytes, but depending on the transport)
>  and consists of three parts:
> @@ -241,9 +280,8 @@ required for the virtqueue according to the following formula:
>  	          + ALIGN(sizeof(u16)*3 + sizeof(struct vring_used_elem)*qsz);
>  	}
>  
> -This currently wastes some space with padding, but also allows future
> -extensions such as the VIRTIO_RING_F_EVENT_IDX extension.  The
> -virtqueue layout structure looks like this:
> +This wastes some space with padding.
> +The legacy virtqueue layout structure therefore looks like this:
>  
>  	struct vring {
>  		// The actual descriptors (16 bytes each)
> @@ -814,9 +852,11 @@ This is done as follows, for each virtqueue a device has:
>    always a power of 2. This controls how big the virtqueue is
>    (see "2.1.4. Virtqueues"). If this field is 0, the virtqueue does not exist.
>  
> -3. Allocate and zero virtqueue in contiguous physical memory, on
> -  a 4096 byte alignment. Write the physical address, divided by
> -  4096 to the Queue Address field.[6]
> +3. Optionally, select a smaller virtqueue size and write it in the Queue Size
> +   field.
> +
> +3. Allocate and zero Descriptor Table, Available and Used rings for the
> +   virtqueue in contiguous physical memory.
>  
>  4. Optionally, if MSI-X capability is present and enabled on the
>    device, select a vector to use to request interrupts triggered

We should move most of this out to the PCI-specific section.  For
example, mmio has a QueueNumMax field instead, and ccw doesn't seem to
have a way of negotiating.

Cheers,
Rusty.