RFC: Doorbell suppression, packed-ring mode and hardware offload

[David Riddoch <driddoch@solarflare.com>]

All,

I'd like to propose a small extension to the packed virtqueue mode. My 
proposal is to add an offset/wrap field, written by the driver, 
indicating how many available descriptors have been added to the ring.

The reason for wanting this is to improve performance of hardware 
devices. Because of high read latency over a PCIe bus, it is important 
for hardware devices to read multiple ring entries in parallel. It is 
desirable to know how many descriptors are available prior to issuing 
these reads, else you risk fetching descriptors that are not yet 
available. As well as wasting bus bandwidth this adds complexity.

I'd previously hoped that VIRTIO_F_NOTIFICATION_DATA would solve this 
problem, but we still have a problem. If you rely on doorbells to tell 
you how many descriptors are available, then you have to keep doorbells 
enabled at all times. This can result in a very high rate of doorbells 
with some drivers, which can become a severe bottleneck (because x86 
CPUs can't emit MMIOs at very high rates).

The proposed offset/wrap field allows devices to disable doorbells when 
appropriate, and determine the latest fill level via a PCIe read.

I suggest the best place to put this would be in the driver area, 
immediately after the event suppression structure.

Presumably we would like this to be an optional feature, as 
implementations of packed mode already exist in the wild. How about 
VIRTIO_F_RING_PACKED_AVAIL_IDX?

If I prepare a patch to the spec is there still time to get this into v1.1?

Best,
David

--
David Riddoch  <driddoch@solarflare.com> -- Chief Architect, Solarflare