RE: virtio-device in µC of a SoC

["Gittinger Joerg (XC-ECO/ESH2)" <Joerg.Gittinger@de.bosch.com>]

Hi,

> I have a concrete request for a use-case to support by virtio, but am not sure where and how to discuss it. 
> So please let me know if this mailing list is not the proper place.

Maybe making a concrete proposal is a better way to handle such a proposal.
Please let me know if this is appropriate or not - just give me some feedback, sort of either
"Go away, we'll never support something like this" or
"Yes, this use case sounds valid. Let us refine/see how we could support this".

Proposal about: 
---------------

Extending virtio-mmio transport by explicit synchronization.

Use case: 
---------

If virtio device is implemented on a different CPU than virtio driver and 
accesses to MMIO region by the driver cannot be "trapped" by the device, 
the need for additional synchronization between virtio driver and device 
is given.
The current virtio MMIO transport specification implicitly expects especially 
during device initialization that virtio device synchronously responds to 
interactive MMIO register write operations by the driver.
In case of a device implemented by a hypervisor this means the hypervisor 
traps accesses to the MMIO region of the driver implemented in a guest.
In case of a device implemented on a different CPU this does not work out 
anymore. For such a case dedicated device initialization steps require 
additional synchronization of MMIO write operations by the driver.

Design proposal:
----------------

1. Introduce a new device feature bit indicating requirement for 
   synchronization when using the MMIO transport.

   E.g. VIRTIO_F_MMIO_SYNCHRONIZE. This feature bit indicates that the device
   requires MMIO synchronization. It must be defined in the first 32 feature 
   bits of the device. 

2. Driver requirements for VIRTIO_F_MMIO_SYNCHRONIZE:

   a. The driver must read the first 32 device feature bits at first in order
      to identify if MMIO access must be synchronized.
   b. In case VIRTIO_F_MMIO_SYNCHRONIZE is indicated by the device, the
      driver must wait for the synchronization acknowledgement by the device
      after writing to: 
        DeviceFeaturesSel
        DriverFeaturesSel
        DriverFeatures
        QueueSel
        Status
        QueueDescHigh
        QueueDriverHigh
        QueueDeviceHigh
        SHMSel
        SHMLenHigh
        SHMBaseHigh
        QueueReset

3. The MMIO synchronization

   The MMIO synchronization is realized by 2 additional registers in the
   MMIO region:

        DriverSyncWatermark (RW)
        DeviceSyncWatermark (R)

   The synchronization works this way: 
   Whenever the driver has written to the registers listed in section 2. 
   above, it increments the value in DriverSyncWatermark by one.
   The driver does not continue with reading/writing any MMIO registers 
   before the device has set the DeviceSyncWatermark to the same value as 
   in DriverSyncWatermark. By updating DeviceSyncWatermark to the same
   value as DriverSyncWatermark the device indicates acknowledgement of
   the change to MMIO registers by the driver.


4. Issues

   - Driver initiated device reset at start of device initialization:

     According to the specification, the driver must reset the device
     before reading the DeviceFeatures. This is before it knows that
     the device needs synchronization.
     In this case the driver must restart the device initialization,
     i.e. reinitiate device reset, this time waiting for 
     acknowledgement of the device.

   - driver -> device notification of queue updates, i.e. QueueNotify:

     If the driver would have to wait for device acknowledgement of 
     updates to QueueNotify after device initialization phase, this
     would have signification impact on the performance and would 
     probably be not accepted by users. Therefore the requirement for 
     synchronization on updates to QueueNotify is dropped.
     Instead, the device is supposed to periodically check for updates 
     on QueueNotify or check the physical memory location of the
     driver queues directly for updates.

Regards,

Joerg Gittinger