Re: [virtio-dev] RFC: Doorbell suppression, packed-ring mode and hardware offload

["Michael S. Tsirkin" <mst@redhat.com>]

On Tue, Feb 12, 2019 at 04:47:08PM +0000, David Riddoch wrote:
> 
> > > > > I would prefer to have the write barrier before writing the idx.
> > > > Well that's driver overhead for something device might never utilise in
> > > > a given workload. If we are optimizing let's optimize for speed.
> > > I think doing the barrier before writing idx is best for speed (see below).
> > I don't see it below :(
> 
> Sorry, I'm not being clear.  If the write barrier is before the idx, then a
> PV device can read the idx, do a single rmb and read a whole bunch of
> descriptors.  As things stand today a PV device has to do an rmb for each
> descriptor that it reads.
> 
> I'm not sure, but this may be what Jason meant when he said "prefetch".

Oh. Right. So for PV it's easy to convert an rmb to a data dependency
instead.  It remains to be seen whether an extra cache miss per
batch is cheaper or more expensive than such a dependency
per descriptor.

I hope Jason can experiment and let us know.


> > > But it is really desirable for complexity: Cases like this are easy to
> > > handle in software, but much much harder in pipelined hardware
> > > implementations.
> > Maybe we should use a flag in event suppression structure.
> > This way device can switch between being notification driven
> > and being driven by index reads.
> 
> On first thought it seems hard to avoid races: On receiving a doorbell the
> device wishes to transition from doorbells to polling, so driver->avail_idx
> currently has an old value; device writes device->flags; a little later
> device reads driver->avail_idx.  It might have a new value written by the
> driver, or an old value...the device can't tell.  Needs more thought.


I would say device writes flags and then reads the descriptor
ring. This is exactly the same sequence that needs to happen
when enabling driver notifications.


> > > > I am also wondering: what would the analog of this feature be for split
> > > > rings? We are burning a feature bit, might as well find a good
> > > > use for it.
> > > I don't have any suggestions.  I worry that associating the same bit with a
> > > split-ring feature would create an undesirable coupling: A device offering X
> > > for packed-ring would also necessarily have to implement Y for split-ring
> > > and vice versa (if both ring types are supported).
> > BTW we really need to look more at how can devices support subsets
> > of features. Right now devices can fail FEATURES_OK but drivers do
> > not recover. Some dependencies are defined by spec and drivers
> > can force them but devices can't.
> 
> Completely agree.  In order to limit complexity in hardware implementations
> it would be nice to be able to express "I support X, and X+Y, but not Y by
> itself" and similar.  There is no way to do that today.
> 
> 
> -- 
> David Riddoch  <driddoch@solarflare.com> -- Chief Architect, Solarflare