Re: [virtio-comment] RE: virtio-device in µC of a SoC

[Stefan Hajnoczi <stefanha@redhat.com>]

On Wed, Jun 29, 2022 at 10:01:28AM +0000, Gittinger Joerg (XC-ECO/ESH2) wrote:
> Hi,
> 
> > I have a concrete request for a use-case to support by virtio, but am not sure where and how to discuss it. 
> > So please let me know if this mailing list is not the proper place.
> 
> Maybe making a concrete proposal is a better way to handle such a proposal.
> Please let me know if this is appropriate or not - just give me some feedback, sort of either
> "Go away, we'll never support something like this" or
> "Yes, this use case sounds valid. Let us refine/see how we could support this".

Hi Joerg,
This is the right mailing list. Maybe the people who are interested in
similar use cases are busy right now, but don't be discouraged!

Your proposal sounds like it's within the scope of VIRTIO and makes
sense in specific circumstances. The next step to making this more
concrete is to send a specification patch
(https://github.com/oasis-tcs/virtio-spec/) for review and voting.

> 
> Proposal about: 
> ---------------
> 
> Extending virtio-mmio transport by explicit synchronization.
> 
> Use case: 
> ---------
> 
> If virtio device is implemented on a different CPU than virtio driver and 
> accesses to MMIO region by the driver cannot be "trapped" by the device, 
> the need for additional synchronization between virtio driver and device 
> is given.
> The current virtio MMIO transport specification implicitly expects especially 
> during device initialization that virtio device synchronously responds to 
> interactive MMIO register write operations by the driver.
> In case of a device implemented by a hypervisor this means the hypervisor 
> traps accesses to the MMIO region of the driver implemented in a guest.
> In case of a device implemented on a different CPU this does not work out 
> anymore. For such a case dedicated device initialization steps require 
> additional synchronization of MMIO write operations by the driver.

In Linux there is the VIRTIO-based rpmsg inter-processor messaging bus:
https://docs.kernel.org/staging/rpmsg.html

It's not developed as part of the VIRTIO spec and unfortunately I don't
know much about how it works, but it came to mind when I read about your
usecase. I believe it does not notifications (interrupts) though, so
maybe your use case is different.

> 
> Design proposal:
> ----------------
> 
> 1. Introduce a new device feature bit indicating requirement for 
>    synchronization when using the MMIO transport.
> 
>    E.g. VIRTIO_F_MMIO_SYNCHRONIZE. This feature bit indicates that the device
>    requires MMIO synchronization. It must be defined in the first 32 feature 
>    bits of the device. 
> 
> 2. Driver requirements for VIRTIO_F_MMIO_SYNCHRONIZE:
> 
>    a. The driver must read the first 32 device feature bits at first in order
>       to identify if MMIO access must be synchronized.
>    b. In case VIRTIO_F_MMIO_SYNCHRONIZE is indicated by the device, the
>       driver must wait for the synchronization acknowledgement by the device
>       after writing to: 
>         DeviceFeaturesSel
>         DriverFeaturesSel
>         DriverFeatures
>         QueueSel
>         Status
>         QueueDescHigh
>         QueueDriverHigh
>         QueueDeviceHigh
>         SHMSel
>         SHMLenHigh
>         SHMBaseHigh
>         QueueReset
> 
> 3. The MMIO synchronization
> 
>    The MMIO synchronization is realized by 2 additional registers in the
>    MMIO region:
> 
>         DriverSyncWatermark (RW)
>         DeviceSyncWatermark (R)
> 
>    The synchronization works this way: 
>    Whenever the driver has written to the registers listed in section 2. 
>    above, it increments the value in DriverSyncWatermark by one.
>    The driver does not continue with reading/writing any MMIO registers 
>    before the device has set the DeviceSyncWatermark to the same value as 
>    in DriverSyncWatermark. By updating DeviceSyncWatermark to the same
>    value as DriverSyncWatermark the device indicates acknowledgement of
>    the change to MMIO registers by the driver.

On some CPU architectures there are memory wait instructions for
monitoring the contents of a memory location. DriverSyncWatermark and
DeviceSyncWatermark look like ideal targets for this type of CPU
instruction.

> 4. Issues
> 
>    - Driver initiated device reset at start of device initialization:
> 
>      According to the specification, the driver must reset the device
>      before reading the DeviceFeatures. This is before it knows that
>      the device needs synchronization.
>      In this case the driver must restart the device initialization,
>      i.e. reinitiate device reset, this time waiting for 
>      acknowledgement of the device.

This sounds fine. It doesn't stop your approach from working.

> 
>    - driver -> device notification of queue updates, i.e. QueueNotify:
> 
>      If the driver would have to wait for device acknowledgement of 
>      updates to QueueNotify after device initialization phase, this
>      would have signification impact on the performance and would 
>      probably be not accepted by users. Therefore the requirement for 
>      synchronization on updates to QueueNotify is dropped.
>      Instead, the device is supposed to periodically check for updates 
>      on QueueNotify or check the physical memory location of the
>      driver queues directly for updates.

The queue index or VIRTIO_F_NOTIFICATION_DATA value may be overwritten
by the driver by the time the device sees it. If the device needs to
poll anyway then it might as well poll the vring. QueueNotify can be
skipped entirely and is unnecessary with VIRTIO_F_MMIO_SYNCHRONIZE.

How do Used Buffer Notifications (typically device->driver interrupts)
work? Are they skipped too and the driver polls the vrings?

Stefan

Attachment: signature.asc
Description: PGP signature