Re: [virtio] [OASIS Issue Tracker] Updated: (VIRTIO-11) Method to stop activity on a queue?

[Cornelia Huck <cornelia.huck@de.ibm.com>]

On Mon, 28 Oct 2013 13:27:17 +1030
Rusty Russell <rusty@au1.ibm.com> wrote:

> Cornelia Huck <cornelia.huck@de.ibm.com> writes:
> > On Fri, 25 Oct 2013 13:16:01 +1030
> > Rusty Russell <rusty@au1.ibm.com> wrote:
> >
> >> And it could be done for MMIO by setting QueuePFN to zero, though the
> >> spec should specify that as allowed on a live device.
> >> 
> >> I'm not sure the best way to add this functionality to CCW.  Cornelia?
> >
> > I'm not quite sure what the requirements are here.
> >
> > - Allow the guest to deregister a single vq? The SET_VQ command should
> >   be able to do that; we would need to make it amply clear that this can
> >   be done in-flight, though.
> > - Allow the guest to tell the host to leave a queue alone for a time?
> >   The best approach for that would probably be a new command (or a
> >   hypercall) passing a queue-active mask; this would allow the guest to
> >   re-enable queue usage later on.
> > - Or the reverse: Allow the host to temporarily halt processing.
> >   Probably with unsolicited interrupts indicating a state change,
> >   similar to what storage servers do. This would be per-device, however.
> 
> Yes, the guest would tell the host to stop looking at the queue.
> 
> For example, the Linux virtio net driver currently is a bit weird.
> Network drivers have init like any other driver, but they have open and
> stop callbacks, for when the network layers want to access them.
> 
> We'd normally allocate receive bufs in open and release them in stop,
> but we can't get our buffers back without resetting the device.
> 
> Similarly, when we go from multiqueue to single queue, it'd be nice to
> be able to free up those buffers.
> 
> It generally seems like a nice feature to have, to stop the host looking
> at a queue while we do arbitrary changes.

Well, I'm not sure about 'arbitrary changes', but we could allow the
following for ccw devices, which should cover the networking use cases:

- Allow to mask/unmask indicators in-flight via a new command.
- Allow to (de)register virtqueues in-flight via the SET_VQ command.

The host needs to make sure that there is no indicator without a
corresponding virtqueue. IOW, it must reject un-masking of indicators
without the matching virtqueue and also reject deregistration of
virtqueues with an active indicator.

We need to be careful, though, to handle host->guest notification vs.
indicator masking races. For classic subchannel I/O, this is not a
problem as a start subchannel on a subchannel that has just become
pending with the status from the notification is always rejected with
a (possibly deferred) condition code as by the architecture. But for
adapter interrupts, the notification is not associated with a
subchannel. This means, theoretically, that the driver may get a
notification while it tries to mask an indicator. The device should be
able, however, to avoid this once it has completed the command. Not a
big deal, just extra care is needed.

Does this match what pci and mmio could do, i.e. activate/deactivate
virtqueues in-flight? Or did I misunderstand?

> 
> Hope that clarifies,
> Rusty.