[PATCH 2/2] virtio-fs: add notification queue

[Stefan Hajnoczi <stefanha@redhat.com>]

The FUSE protocol allows the file server (device) to initiate
communication with the client (driver) using FUSE notify messages.
Normally only the client can initiate communication.  This feature is
used to report asynchronous events that are not related to an in-flight
request.

This patch adds a notification queue that works like an rx queue in
other types of device.  The device can emit FUSE notify messages by
using a buffer from this queue.

This mechanism was designed by Vivek Goyal <vgoyal@redhat.com>.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
TODO:
 * Flow control?  What happens when the notification queue is empty when
   the device wants to emit a message.  We cannot assume unlimited
   device resources for holding back pending notifications.  Eventually
   they will have to be dropped and/or the device needs to report an
   error.
---
 conformance.tex |  1 +
 virtio-fs.tex   | 63 +++++++++++++++++++++++++++++++++++++++++--------
 2 files changed, 54 insertions(+), 10 deletions(-)

diff --git a/conformance.tex b/conformance.tex
index 25d11ec..a43eea3 100644
--- a/conformance.tex
+++ b/conformance.tex
@@ -190,6 +190,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \begin{itemize}
 \item \ref{drivernormative:Device Types / File System Device / Device configuration layout}
 \item \ref{drivernormative:Device Types / File System Device / Device Operation / Device Operation: High Priority Queue}
+\item \ref{drivernormative:Device Types / File System Device / Device Operation / Device Operation: Notification Queue}
 \item \ref{drivernormative:Device Types / File System Device / Device Operation / Device Operation: DAX Window}
 \end{itemize}
 
diff --git a/virtio-fs.tex b/virtio-fs.tex
index 158d066..a94a377 100644
--- a/virtio-fs.tex
+++ b/virtio-fs.tex
@@ -25,24 +25,33 @@ \subsection{Virtqueues}\label{sec:Device Types / File System Device / Virtqueues
 
 \begin{description}
 \item[0] hiprio
-\item[1\ldots n] request queues
+\item[1] notification queue
+\item[2\ldots n] request queues
 \end{description}
 
+The notification queue only exists if VIRTIO_FS_F_NOTIFICATION is set.
+
 \subsection{Feature bits}\label{sec:Device Types / File System Device / Feature bits}
 
-There are currently no feature bits defined.
+\begin{description}
+\item[VIRTIO_FS_F_NOTIFICATION (0)] Device has support for FUSE notify
+    messages.  The notification queue is virtqueue 1.
+\end{description}
 
 \subsection{Device configuration layout}\label{sec:Device Types / File System Device / Device configuration layout}
 
-All fields of this configuration are always available.
-
 \begin{lstlisting}
 struct virtio_fs_config {
         char tag[36];
         le32 num_request_queues;
+        le32 notify_buf_size;
 };
 \end{lstlisting}
 
+The \field{tag} and \field{num_request_queues} fields are always available.
+The \field{notify_buf_size} field is only available when
+VIRTIO_FS_F_NOTIFICATION is set.
+
 \begin{description}
 \item[\field{tag}] is the name associated with this file system.  The tag is
     encoded in UTF-8 and padded with NUL bytes if shorter than the
@@ -53,6 +62,8 @@ \subsection{Device configuration layout}\label{sec:Device Types / File System De
     there are no ordering guarantees between requests made available on
     different queues.  Use of multiple queues is intended to increase
     performance.
+\item[\field{notify_buf_size}] is the minimum number of bytes required for each
+    buffer in the notification queue.
 \end{description}
 
 \drivernormative{\subsubsection}{Device configuration layout}{Device Types / File System Device / Device configuration layout}
@@ -65,13 +76,20 @@ \subsection{Device configuration layout}\label{sec:Device Types / File System De
 
 The device MUST set \field{num_request_queues} to 1 or greater.
 
+The device MUST set \field{notify_buf_size} to be large enough to hold any of
+the FUSE notify messages that this device emits.
+
 \subsection{Device Initialization}\label{Device Types / File System Device / Device Initialization}
 
-On initialization the driver first discovers the device's virtqueues.  The FUSE
-session is started by sending a FUSE\_INIT request as defined by the FUSE
-protocol on one request virtqueue.  All virtqueues provide access to the same
-FUSE session and therefore only one FUSE\_INIT request is required regardless
-of the number of available virtqueues.
+On initialization the driver first discovers the device's virtqueues.
+
+The driver populates the notification queue with buffers for receiving FUSE
+notify messages if VIRTIO_FS_F_NOTIFICATION is set.
+
+The FUSE session is started by sending a FUSE\_INIT request as defined by the
+FUSE protocol on one request virtqueue.  All virtqueues provide access to the
+same FUSE session and therefore only one FUSE\_INIT request is required
+regardless of the number of available virtqueues.
 
 \subsection{Device Operation}\label{sec:Device Types / File System Device / Device Operation}
 
@@ -88,7 +106,8 @@ \subsection{Device Operation}\label{sec:Device Types / File System Device / Devi
       full.
 \end{itemize}
 
-Note that FUSE notification requests are not supported.
+FUSE notify messages are received on the notification queue if
+VIRTIO_FS_F_NOTIFICATION is set.
 
 \subsubsection{Device Operation: Request Queues}\label{sec:Device Types / File System Device / Device Operation / Device Operation: Request Queues}
 
@@ -179,6 +198,30 @@ \subsubsection{Device Operation: High Priority Queue}\label{sec:Device Types / F
 
 The driver MUST anticipate that request queues are processed concurrently with the hiprio queue.
 
+\subsubsection{Device Operation: Notification Queue}\label{sec:Device Types / File System Device / Device Operation / Device Operation: Notification Queue}
+
+The notification queue is populated with buffers by the driver and these
+buffers are used by the device to emit FUSE notify messages.  Notification
+queue buffer layout is as follows:
+
+\begin{lstlisting}
+struct virtio_fs_notify {
+        // Device-writable part
+        struct fuse_out_header out_hdr;
+        char outarg[notify_buf_size - sizeof(struct fuse_out_header)];
+};
+\end{lstlisting}
+
+\field{outarg} contains the FUSE notify message payload that depends on the
+type of notification being emitted.
+
+\drivernormative{\paragraph}{Device Operation: Notification Queue}{Device Types / File System Device / Device Operation / Device Operation: Notification Queue}
+
+The driver MUST provide buffers of at least \field{notify_buf_size} bytes.
+
+The driver SHOULD replenish notification queue buffers sufficiently quickly so
+that there is always at least one available buffer.
+
 \subsubsection{Device Operation: DAX Window}\label{sec:Device Types / File System Device / Device Operation / Device Operation: DAX Window}
 
 FUSE\_READ and FUSE\_WRITE requests transfer file contents between the
-- 
2.23.0