Re: [virtio] [PATCH v7 08/11] packed virtqueues: more efficient virtqueue layout

[Cornelia Huck <cohuck@redhat.com>]

On Tue, 23 Jan 2018 02:01:07 +0200
"Michael S. Tsirkin" <mst@redhat.com> wrote:

> Performance analysis of this is in my kvm forum 2016 presentation.  The
> idea is to have a r/w descriptor in a ring structure, replacing the used
> and available ring, index and descriptor buffer.
> 
> This is also easier for devices to implement than the 1.0 layout.
> Several more enhancements will be necessary to actually make this
> efficient for devices to use.
> 
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  content.tex     |  25 ++-
>  packed-ring.tex | 678 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 700 insertions(+), 3 deletions(-)
>  create mode 100644 packed-ring.tex

(...)

> +\subsubsection{Driver notifications}
> +\label{sec:Packed Virtqueues / Driver notifications}
> +Whenever not suppressed by Device Event Suppression,
> +driver is required to notify the device after
> +making changes to the virtqueue.
> +
> +Some devices benefit from ability to find out the number of
> +available descriptors in the ring, and whether to send
> +interrupts to drivers without accessing virtqueue in memory:
> +for efficiency or as a debugging aid.
> +
> +To help with these optimizations, driver notifications
> +to the device include the following information:
> +
> +\begin{itemize}
> +\item VQ number
> +\item Offset (in units of descriptor size) within the ring
> +      where the next available descriptor will be written
> +\item Wrap Counter referring to the next available
> +      descriptor
> +\end{itemize}
> +
> +Note that driver can trigger multiple notifications even without
> +making any more changes to the ring. These would then have
> +identical \field{Offset} and \field{Wrap Counter} values.

(...)

> +\subsection{Driver Notification Format}\label{sec:Basic
> +Facilities of a Virtio Device / Packed Virtqueues / Driver Notification Format}
> +
> +The following structure is used to notify device of
> +device events - i.e. available descriptors:
> +
> +\begin{lstlisting}
> +__le16 vqn;
> +__le16 next_off : 15;
> +int    next_wrap : 1;
> +\end{lstlisting}

(...)

> +\subsubsection{Notifying The Device}\label{sec:Basic Facilities
> +of a Virtio Device / Packed Virtqueues / Supplying Buffers to The Device / Notifying The Device}
> +
> +The actual method of device notification is bus-specific, but generally
> +it can be expensive.  So the device MAY suppress such notifications if it
> +doesn't need them, using the Driver Event Suppression structure
> +as detailed in section \ref{sec:Basic
> +Facilities of a Virtio Device / Packed Virtqueues / Event
> +Suppression Structure Format}.
> +
> +The driver has to be careful to expose the new \field{flags}
> +value before checking if notifications are suppressed.

This is all I could find regarding notifications, and it leaves me
puzzled how notifications are actually supposed to work; especially,
where that driver notification structure is supposed to be relayed.

I'm obviously coming from a ccw perspective, but I don't think that pci
is all that different (well, hopefully).

Up to now, we notified for a certain virtqueue -- i.e., the device
driver notified the device that there is something to process for a
certain queue. ccw uses the virtqueue number in a gpr for a hypercall,
pci seems to use a write to the config space IIUC. With the packed
layout, we have more payload per notification. We should be able to put
it in the same gpr than the virtqueue for ccw (if needed, with some
compat magic, or with a new hypercall, which would be ugly but doable).
Not sure how this is supposed to work with pci.

Has there been any prototyping done to implement this in qemu + KVM?
I'm unsure how this will work with ioeventfds, which just trigger.