Re: [PATCH 2/2] virtio-fs: add notification queue

["Dr. David Alan Gilbert" <dgilbert@redhat.com>]

* Stefan Hajnoczi (stefanha@redhat.com) wrote:
> The FUSE protocol allows the file server (device) to initiate
> communication with the client (driver) using FUSE notify messages.
> Normally only the client can initiate communication.  This feature is
> used to report asynchronous events that are not related to an in-flight
> request.
> 
> This patch adds a notification queue that works like an rx queue in
> other types of device.  The device can emit FUSE notify messages by
> using a buffer from this queue.
> 
> This mechanism was designed by Vivek Goyal <vgoyal@redhat.com>.
> 
> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> ---
> TODO:
>  * Flow control?  What happens when the notification queue is empty when
>    the device wants to emit a message.  We cannot assume unlimited
>    device resources for holding back pending notifications.  Eventually
>    they will have to be dropped and/or the device needs to report an
>    error.
> ---
>  conformance.tex |  1 +
>  virtio-fs.tex   | 63 +++++++++++++++++++++++++++++++++++++++++--------
>  2 files changed, 54 insertions(+), 10 deletions(-)
> 
> diff --git a/conformance.tex b/conformance.tex
> index 25d11ec..a43eea3 100644
> --- a/conformance.tex
> +++ b/conformance.tex
> @@ -190,6 +190,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
>  \begin{itemize}
>  \item \ref{drivernormative:Device Types / File System Device / Device configuration layout}
>  \item \ref{drivernormative:Device Types / File System Device / Device Operation / Device Operation: High Priority Queue}
> +\item \ref{drivernormative:Device Types / File System Device / Device Operation / Device Operation: Notification Queue}
>  \item \ref{drivernormative:Device Types / File System Device / Device Operation / Device Operation: DAX Window}
>  \end{itemize}
>  
> diff --git a/virtio-fs.tex b/virtio-fs.tex
> index 158d066..a94a377 100644
> --- a/virtio-fs.tex
> +++ b/virtio-fs.tex
> @@ -25,24 +25,33 @@ \subsection{Virtqueues}\label{sec:Device Types / File System Device / Virtqueues
>  
>  \begin{description}
>  \item[0] hiprio
> -\item[1\ldots n] request queues
> +\item[1] notification queue
> +\item[2\ldots n] request queues
>  \end{description}
>  
> +The notification queue only exists if VIRTIO_FS_F_NOTIFICATION is set.
> +
>  \subsection{Feature bits}\label{sec:Device Types / File System Device / Feature bits}
>  
> -There are currently no feature bits defined.
> +\begin{description}
> +\item[VIRTIO_FS_F_NOTIFICATION (0)] Device has support for FUSE notify
> +    messages.  The notification queue is virtqueue 1.
> +\end{description}
>  
>  \subsection{Device configuration layout}\label{sec:Device Types / File System Device / Device configuration layout}
>  
> -All fields of this configuration are always available.
> -
>  \begin{lstlisting}
>  struct virtio_fs_config {
>          char tag[36];
>          le32 num_request_queues;
> +        le32 notify_buf_size;
>  };
>  \end{lstlisting}
>  
> +The \field{tag} and \field{num_request_queues} fields are always available.
> +The \field{notify_buf_size} field is only available when
> +VIRTIO_FS_F_NOTIFICATION is set.
> +
>  \begin{description}
>  \item[\field{tag}] is the name associated with this file system.  The tag is
>      encoded in UTF-8 and padded with NUL bytes if shorter than the
> @@ -53,6 +62,8 @@ \subsection{Device configuration layout}\label{sec:Device Types / File System De
>      there are no ordering guarantees between requests made available on
>      different queues.  Use of multiple queues is intended to increase
>      performance.
> +\item[\field{notify_buf_size}] is the minimum number of bytes required for each
> +    buffer in the notification queue.
>  \end{description}
>  
>  \drivernormative{\subsubsection}{Device configuration layout}{Device Types / File System Device / Device configuration layout}
> @@ -65,13 +76,20 @@ \subsection{Device configuration layout}\label{sec:Device Types / File System De
>  
>  The device MUST set \field{num_request_queues} to 1 or greater.
>  
> +The device MUST set \field{notify_buf_size} to be large enough to hold any of
> +the FUSE notify messages that this device emits.
> +

Is there some ordering requirement here to ensure that the device has
set notify_buf_size before the driver starts trying to populate the
queue with empties?

Dave

>  \subsection{Device Initialization}\label{Device Types / File System Device / Device Initialization}
>  
> -On initialization the driver first discovers the device's virtqueues.  The FUSE
> -session is started by sending a FUSE\_INIT request as defined by the FUSE
> -protocol on one request virtqueue.  All virtqueues provide access to the same
> -FUSE session and therefore only one FUSE\_INIT request is required regardless
> -of the number of available virtqueues.
> +On initialization the driver first discovers the device's virtqueues.
> +
> +The driver populates the notification queue with buffers for receiving FUSE
> +notify messages if VIRTIO_FS_F_NOTIFICATION is set.
> +
> +The FUSE session is started by sending a FUSE\_INIT request as defined by the
> +FUSE protocol on one request virtqueue.  All virtqueues provide access to the
> +same FUSE session and therefore only one FUSE\_INIT request is required
> +regardless of the number of available virtqueues.
>  
>  \subsection{Device Operation}\label{sec:Device Types / File System Device / Device Operation}
>  
> @@ -88,7 +106,8 @@ \subsection{Device Operation}\label{sec:Device Types / File System Device / Devi
>        full.
>  \end{itemize}
>  
> -Note that FUSE notification requests are not supported.
> +FUSE notify messages are received on the notification queue if
> +VIRTIO_FS_F_NOTIFICATION is set.
>  
>  \subsubsection{Device Operation: Request Queues}\label{sec:Device Types / File System Device / Device Operation / Device Operation: Request Queues}
>  
> @@ -179,6 +198,30 @@ \subsubsection{Device Operation: High Priority Queue}\label{sec:Device Types / F
>  
>  The driver MUST anticipate that request queues are processed concurrently with the hiprio queue.
>  
> +\subsubsection{Device Operation: Notification Queue}\label{sec:Device Types / File System Device / Device Operation / Device Operation: Notification Queue}
> +
> +The notification queue is populated with buffers by the driver and these
> +buffers are used by the device to emit FUSE notify messages.  Notification
> +queue buffer layout is as follows:
> +
> +\begin{lstlisting}
> +struct virtio_fs_notify {
> +        // Device-writable part
> +        struct fuse_out_header out_hdr;
> +        char outarg[notify_buf_size - sizeof(struct fuse_out_header)];
> +};
> +\end{lstlisting}
> +
> +\field{outarg} contains the FUSE notify message payload that depends on the
> +type of notification being emitted.
> +
> +\drivernormative{\paragraph}{Device Operation: Notification Queue}{Device Types / File System Device / Device Operation / Device Operation: Notification Queue}
> +
> +The driver MUST provide buffers of at least \field{notify_buf_size} bytes.
> +
> +The driver SHOULD replenish notification queue buffers sufficiently quickly so
> +that there is always at least one available buffer.
> +
>  \subsubsection{Device Operation: DAX Window}\label{sec:Device Types / File System Device / Device Operation / Device Operation: DAX Window}
>  
>  FUSE\_READ and FUSE\_WRITE requests transfer file contents between the
> -- 
> 2.23.0
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK