[PATCH RFC] Introduce multiple interrupt support (config + per queue) for MMIO-based devices.

[Sergey Viktorov <sergeyv@parallels.com>]

This is an attempt to remove performance handicap of MMIO-based devices induced
by single interrupt and mandatory acknowledgement limitations. While PCI-based
devices can take advantage of MSI-X interrupts, MMIO part of the spec relies on
classic status/acknowledgement pattern and made no provisions for using multiple
interrupts.
The suggestion is to add interrupt assignment registers similar to configuration
fields of a PCI-based device that would allow configuration and queue device
notifications to use separate interrupts. Set of interrupts usable by the device
is defined by the virtual platform and declared via DeviceTree and/or ACPI DSDT
the same way it's done now for single interrupt. The new device capability is
exposed through VIRTIO_F_MMIO_MULTI_IRQ feature bit. If the feature is negotiated
the device makes new registers available and also disables mandatory interrupt
acknowledgement semantics.

Our primary motivation is to implement an ARM-based virtual platform that does
not expose PCI bus and saves on associated code complexity. This is perfectly
doable right now but performance critical devices, namely network and block, are
held back by interrupt handling semantics in comparison to their PCI-based twins.
---
 content.tex | 38 +++++++++++++++++++++++++++++++++++---
 1 file changed, 35 insertions(+), 3 deletions(-)

diff --git a/content.tex b/content.tex
index 620c0e2..0dc1e48 100644
--- a/content.tex
+++ b/content.tex
@@ -1753,6 +1753,14 @@ \subsection{MMIO Device Register Layout}\label{sec:Virtio Transport Options / Vi
     bits accessible by writing to \field{DriverFeatures}.
   }
   \hline 
+  \mmioreg{ConfigInt}{Configuration interrupt index}{0x028}{W}{%
+    Assign one of device interrupts to configuration change. Device
+    interrupt indices start at zero and follow DeviceTree and/or
+    ACPI DSDT declaration order. The default interrupt index is 0.
+    This register may only be used if VIRTIO_F_MMIO_MULTI_IRQ feature
+    has been negotiated.
+  }
+  \hline 
   \mmioreg{QueueSel}{Virtual queue index}{0x030}{W}{%
     Writing to this register selects the virtual queue that the
     following operations on \field{QueueNumMax}, \field{QueueNum}, \field{QueueReady},
@@ -1775,6 +1783,15 @@ \subsection{MMIO Device Register Layout}\label{sec:Virtio Transport Options / Vi
     writing to \field{QueueSel}.
   }
   \hline 
+  \mmioreg{QueueInt}{Virtual queue interrupt index}{0x03c}{W}{%
+    Assign one of device interrupts to queue previously selected
+    by writing to \field{QueueSel}. Device interrupt indices start at
+    zero and follow DeviceTree and/or ACPI DSDT declaration order.
+    By default all queues have interrupt 0 assigned to them.
+    This register may only be used if VIRTIO_F_MMIO_MULTI_IRQ feature
+    has been negotiated.
+  }
+  \hline 
   \mmioreg{QueueReady}{Virtual queue ready bit}{0x044}{RW}{%
     Writing one (0x1) to this register notifies the device that it can
     execute requests from this virtual queue. Reading from this register
@@ -1809,12 +1826,14 @@ \subsection{MMIO Device Register Layout}\label{sec:Virtio Transport Options / Vi
       \item [Configuration Change Notification] - bit 1 - the interrupt was
         asserted because the configuration of the device has changed.
     \end{description}
+    This register is not used if VIRTIO_F_MMIO_MULTI_IRQ is negotiated.
   }
   \hline 
   \mmioreg{InterruptACK}{Interrupt acknowledge}{0x064}{W}{%
     Writing a value with bits set as defined in \field{InterruptStatus}
     to this register notifies the device that events causing
     the interrupt have been handled.
+    This register is not used if VIRTIO_F_MMIO_MULTI_IRQ is negotiated.
   }
   \hline 
   \mmioreg{Status}{Device status}{0x070}{RW}{%
@@ -1948,8 +1967,9 @@ \subsection{MMIO Device Register Layout}\label{sec:Virtio Transport Options / Vi
 
 The driver MUST ignore undefined bits in \field{InterruptStatus}.
 
-The driver MUST write a value with a bit mask describing events it handled into \field{InterruptACK} when
-it finishes handling an interrupt and MUST NOT set any of the undefined bits in the value.
+If VIRTIO_F_MMIO_MULTI_IRQ is not negotiated the driver MUST write a value with a bit mask
+describing events it handled into \field{InterruptACK} when it finishes handling
+an interrupt and MUST NOT set any of the undefined bits in the value.
 
 \subsection{MMIO-specific Initialization And Device Operation}\label{sec:Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation}
 
@@ -2016,12 +2036,20 @@ \subsubsection{Available Buffer Notifications}\label{sec:Virtio Transport Option
 
 \subsubsection{Notifications From The Device}\label{sec:Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation / Notifications From The Device}
 
-The memory mapped virtio device is using a single, dedicated
+By default the memory mapped virtio device is using a single, dedicated
 interrupt signal, which is asserted when at least one of the
 bits described in the description of \field{InterruptStatus}
 is set. This is how the device sends a used buffer notification
 or a configuration change notification to the device.
 
+On platforms where multiple edge triggered or message signalled
+interrupts per device are supported the device may advertise
+VIRTIO_F_MMIO_MULTI_IRQ feature. If negotiated, this feature removes
+mandatory interrupt acknowledgement and allows per queue interrupt
+assignment via \field{ConfigInt} and \field{QueueInt} in a manner
+similar to MSI-X. The list of available device interrupts is
+defined by platform and declared via DeviceTree and/or ACPI DSDT.
+
 \drivernormative{\paragraph}{Notifications From The Device}{Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation / Notifications From The Device}
 After receiving an interrupt, the driver MUST read
 \field{InterruptStatus} to check what caused the interrupt (see the
@@ -6596,6 +6624,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
   transport specific.
   For more details about driver notifications over PCI see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Available Buffer Notifications}.
 
+  \item[VIRTIO_F_MMIO_MULTI_IRQ(40)] Applicable only to MMIO-based devices,
+  this feature removes mandatory interrupt acknowledgement and allows per queue
+  interrupt assignment.
+  See \ref{sec:Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation / Notifications From The Device}
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
-- 
2.24.3 (Apple Git-128)