Re: [virtio-dev] On doorbells (queue notifications)

[Cornelia Huck <cohuck@redhat.com>]

On Wed, 15 Jul 2020 16:47:32 +0100
Stefan Hajnoczi <stefanha@redhat.com> wrote:

> On Wed, Jul 15, 2020 at 02:29:04PM +0100, Alex BennÃe wrote:
> > Stefan Hajnoczi <stefanha@redhat.com> writes:  
> > > On Tue, Jul 14, 2020 at 10:43:36PM +0100, Alex BennÃe wrote:  
> > >> Finally I'm curious if this is just a problem avoided by the s390
> > >> channel approach? Does the use of messages over a channel just avoid the
> > >> sort of bouncing back and forth that other hypervisors have to do when
> > >> emulating a device?  
> > >
> > > What does "bouncing back and forth" mean exactly?  
> > 
> > Context switching between guest and hypervisor.  
> 
> I have CCed Cornelia Huck, who can explain the lifecycle of an I/O
> request on s390 channel I/O.

Having read through this thread, I think this is mostly about
notifications? These are not using channel programs (which are only
used for things like feature negotiation, or emulating reading/writing
a config space, which does not really exist for channel devices.)

First, I/O and interrupts are highly abstracted on s390; much of the
register accesses or writes done on other architectures is just not
seen on s390.

Traditionally, I/O interrupts on s390 are tied to a subchannel; you
have a rather heavyweight process for that:

guest								host

					put status into subchannel
					queue interrupt
open up for I/O interrupt
					store some data into lowcore
					do PSW swap
interrupt handler called
read from lowcore
call tsch for subchannel
					store subchannel status into
					control block
process control block
look at subchannel indicators
virtio queue processing

This is only used for configuration change notifications, or for very
old legacy virtio implementations.

There's an alternative mechanism not tied to a subchannel, called
'adapter interrupts'. (It is even used to implement MSI-X on s390x,
which is why only virtio-pci devices using MSI-X are supported on
s390x.) It uses two-staged indicators: a global indicator to show
whether any secondary indicator is set, and secondary indicators (which
are per virtqueue in the virtio case.)

guest								host

					set queue indicator(s)
					set global indicator
					queue interrupt iff global
					indicator had not been set
open up for I/O interrupt
					store some data into lowcore
					do PSW swap
interrupt handler called
read from lowcore
look at indicators
virtio queue processing

This has less context switches than traditional I/O interrupts; but I
think the main benefit comes from the ability to batch notifications:
as long as the guest is still processing indicators, the host does not
need to notify again, it can just set indicators (which is why the
guest always needs to do two passes at processing.) We can already
batch per-device indicators with the classic approach, but adapter
interrupts allow to batch even across many devices.

Attachment: pgpuSE2XoTVvq.pgp
Description: OpenPGP digital signature