Re: [virtio] [PATCH] virtqueue: flexible layout, size, alignment

["Michael S. Tsirkin" <mst@redhat.com>]

On Fri, Sep 13, 2013 at 10:40:40AM +0930, Rusty Russell wrote:
> "Michael S. Tsirkin" <mst@redhat.com> writes:
> > Transports can now lay out available/used/descriptor
> > regions in a flexible way.
> > This is useful for embedded systems to save memory,
> > and for large systems to reduce the need for
> > physically-contigious memory.
> >
> > This does not add a way to actually program this
> > in any of the transports, so it's not useful by
> > itself, a follow-up patch with add a way to
> > program this for PCI.
> >
> > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > ---
> >  virtio-v1.0-wd01-part1-specification.txt | 52 ++++++++++++++++++++++++++++----
> >  1 file changed, 46 insertions(+), 6 deletions(-)
> >
> > diff --git a/virtio-v1.0-wd01-part1-specification.txt b/virtio-v1.0-wd01-part1-specification.txt
> > index fcd9fd7..9a40973 100644
> > --- a/virtio-v1.0-wd01-part1-specification.txt
> > +++ b/virtio-v1.0-wd01-part1-specification.txt
> > @@ -223,6 +223,45 @@ transmit and one for receive.  Each queue has a 16-bit queue size
> >  parameter, which sets the number of entries and implies the total size
> >  of the queue.
> >  
> > +Each virtqueue consists of three parts:
> > +
> > +	Descriptor Table
> > +	Available Ring
> > +	Used Ring
> > +
> > +where each part is physically-contiguous in guest memory,
> > +and has different alignment requirements.
> > +
> > +The Queue Size field controls the total number of bytes
> > +required for each part of the virtqueue.
> > +
> > +The memory aligment and size requirements, in bytes, of each part of the
> > +virtqueue are summarized in the following table (qsz is the Queue Size field):
> > +
> > ++------------+---------------------------------+
> > +| Virtqueue Part    | Alignment | Size         |
> > ++------------+---------------------------------+
> > ++------------+---------------------------------+
> > +| Descriptor Table  | 16        | 16 * qsz     |
> 
> Alignment 16: isn't that overkill?
> 
> > ++------------+---------------------------------+
> > +| Available Ring    | 2         | 6 + 2 * qsz  |
> > ++------------+---------------------------------+
> > +| Used Ring         | 4         | 6 + 4 * qsz  |
> > ++------------+---------------------------------+
> > +
> > +When the driver wants to send a buffer to the device, it fills in
> > +a slot in the descriptor table (or chains several together), and
> > +writes the descriptor index into the available ring.  It then
> > +notifies the device. When the device has finished a buffer, it
> > +writes the descriptor into the used ring, and sends an interrupt.
> > +
> > +
> > +100.100.4.1. Legacy Interfaces: A Note on Virtqueue Layout
> > +--------------------------------------
> > +
> > +For Legacy Interfaces, several additional
> > +restrictions are placed on the virtqueue layout:
> > +
> >  Each virtqueue occupies two or more physically-contiguous pages
> >  (usually defined as 4096 bytes, but depending on the transport)
> >  and consists of three parts:
> > @@ -241,9 +280,8 @@ required for the virtqueue according to the following formula:
> >  	          + ALIGN(sizeof(u16)*3 + sizeof(struct vring_used_elem)*qsz);
> >  	}
> >  
> > -This currently wastes some space with padding, but also allows future
> > -extensions such as the VIRTIO_RING_F_EVENT_IDX extension.  The
> > -virtqueue layout structure looks like this:
> > +This wastes some space with padding.
> > +The legacy virtqueue layout structure therefore looks like this:
> >  
> >  	struct vring {
> >  		// The actual descriptors (16 bytes each)
> > @@ -814,9 +852,11 @@ This is done as follows, for each virtqueue a device has:
> >    always a power of 2. This controls how big the virtqueue is
> >    (see "2.1.4. Virtqueues"). If this field is 0, the virtqueue does not exist.
> >  
> > -3. Allocate and zero virtqueue in contiguous physical memory, on
> > -  a 4096 byte alignment. Write the physical address, divided by
> > -  4096 to the Queue Address field.[6]
> > +3. Optionally, select a smaller virtqueue size and write it in the Queue Size
> > +   field.
> > +
> > +3. Allocate and zero Descriptor Table, Available and Used rings for the
> > +   virtqueue in contiguous physical memory.
> >  
> >  4. Optionally, if MSI-X capability is present and enabled on the
> >    device, select a vector to use to request interrupts triggered
> 
> We should move most of this out to the PCI-specific section.  For
> example, mmio has a QueueNumMax field instead, and ccw doesn't seem to
> have a way of negotiating.
> 
> Cheers,
> Rusty.

Actually, this is already in the PCI-specific section.
This chunk affects "2.3.1.3.1.2. Virtqueue Configuration".