Virtio Extended Descriptors idea/proposal

[Mario Torrecillas Rodriguez <Mario.Rodriguez@arm.com>]

Hello,

Iâd like to circulate an idea around - if something like this has been discussed in the past, please let me know (I was not able to find anything in the archives).

In the packed virtqueue design, a single descriptor contains a buffer address along with associated metadata (ID and length) as well as a set of flags. All descriptors are associated with a piece of data that is indirectly referenced by the descriptor, and hence the receiver needs to:

• Get the descriptor from memory
• Dereference the buffer address and read the underlying data from memory

This is well suited for cases where data allocation and transmission through the virtqueue take place at different points in time, or when the data will be passed around several times. One example is networking, where packets are allocated at some point, operated on (possibly not even within a single thread), and then eventually sent out to the NIC.

However I think it would be beneficial to also support a scheme where data is directly embedded in the descriptor itself (what I call here âextended descriptorâ), which would make virtio more generically applicable. This works well in cases where the data is generated right before the descriptor is sent - for example, all configuration/control messages, or when the amount of data to be sent is very small and fixed in size. Certain hardware devices might also prefer to operate this way since they donât have the extra level of indirection.

In such cases, simplification benefits aside, we expect this scheme to present a number of performance benefits:

• When using referenced buffers, if the back-end is implemented in hardware with an IOMMU, buffer addresses must be appropriately mapped in advance so that the device can dereference them. This is not required if the data is embedded in the descriptor.
• Small enough extended descriptors could be laid out in memory such that they fit in a single cache line. This would eliminate the need to load the referenced buffer and consume an extra cache line - with a single load you can retrieve all the information.
• Similarly, since an external buffer does not need to be allocated, the TLB should benefit from this.

The intention of this email is to present the idea and see what other people think - I would be happy to work on a patch for the specification if there is interest, and use it to drive a more detailed discussion. In short, I think the following changes would be required:

• Add a device-independent feature bit to convey the fact that the device/driver supports extended descriptors. This feature could be dependent on VIRTIO_F_RING_PACKED.
• Add a per-virtqueue configuration parameter that defines the amount of extended data in the descriptor (either in bytes, or a multiple of some other width, such as 64-bit words or cache lines).
• Add a flag to the descriptor to determine whether data is being referenced by the buffer address or embedded in the extended data portion (note that, even where data is referenced, the extended data portion must still be there, even if unused - the reason is that all descriptors in a given queue should be the same size so as to not prevent batching).

When this feature is negotiated and enabled for a given queue, the size of the descriptor is effectively (16 bytes + extended descriptor size) and drivers/devices are free to insert any device-specific data in the extended portion of the descriptor.

Mario.