Re: [virtio-comment] [PATCH RFC] virtio: introduce VIRTIO_F_DEVICE_STOP

[Jason Wang <jasowang@redhat.com>]

----- Original Message -----
> On Mon, Dec 28, 2020 at 03:01:57PM +0800, Jason Wang wrote:
> > 
> > On 2020/12/28 äå2:21, Halil Pasic wrote:
> > > On Sun, 27 Dec 2020 05:00:05 -0500
> > > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > 
> > > > On Fri, Dec 25, 2020 at 08:38:35AM +0100, Halil Pasic wrote:
> > > > > When driver is trying to set DEVICE_STOPPED, the device MUST not
> > > > > process new avail requests and MUST complete all requests that is
> > > > > currently processing before setting DEVICE_STOPPED.
> > > > ...
> > > > 
> > > > > Since we have a synchronous API setting DEVICE_STOPPED would also
> > > > > have to
> > > > > block until all in-flight requests are completed.
> > > > Judging from the surrounding discussion,
> > > > when you say complete you seem to mean "use", and
> > > My intention was merely to paraphrase Jason's proposal which says:
> > > 
> > > +When driver is trying to set DEVICE_STOPPED, the device MUST not
> > > +process new avail requests and MUST complete all requests that is
> > > +currently processing before setting DEVICE_STOPPED.
> > > 
> > > > I'm not sure how you define in flight, but
> > > > it seems there could be many of these (up to a full queue?)
> > > In the follow-on discussion 'in-flight' emerged as an alternative to
> > > 'all requests that is currently processing' form the proposed text.
> > > 
> > > A large part of the discussion is, IMHO, about finding precise
> > > definitions, for what the quoted paragraph is trying to express.
> > > 
> > > > so waiting for all available buffers to be
> > > > used might indeed require an asynchronous interface,
> > > > which gets complex very quickly.
> > 
> > 
> > Some part of the virtio has enforced an asynchronous interface during
> > reset:
> > 
> > For MMIO the spec said:
> > 
> > """
> > 
> > To stop using the queue the driver MUST write zero (0x0) to this QueueReady
> > and MUST read the value back to ensure synchronization.
> > 
> > """
> > 
> > For PCI it said:
> > 
> > """
> > 
> > After writing 0 to device_status, the driver MUST wait for a read of
> > device_status to return 0 before reinitializing the device.
> > 
> > """
> 
> Absolutely. It's ok for simple things.
> However doing anything major like waiting for IO
> in a tight loop is a problem.

It's mainly used for Qemu. And driver can choose to cancel the stop by
trying to clear the bit. Does this work?


> 
> 
> 
> > 
> > > > 
> > > > However wouldn't it be possible for device to just cancel
> > > > processing available buffers even if it started processing
> > > > them? Some buffers could be in indeterminate state
> > > > (e.g. we might not have a way to know how much data did
> > > > device have time to write into a buffer).
> > > > 
> > > Maybe Jason can answer this one.
> > 
> > 
> > For networking device, it should be possible (packet could be dropped or
> > duplicated). But I'm not sure it works for other device. E.g for the block
> > devices that needs to communicate with a remote backend. Waat's more
> > important, my understadning is that the intermediate state is something
> > that
> > we need to avoid (hard to be migrated anyhow).
> 
> The difficulty is on the device side though, so why not say: if it wants to
> flush IO that's up to it.

Do you mean to introduce a device specific way to flush IO? If yes, it
actually introduces a new intermediate state implicitly:

1) device is stopped
2) device is stopped and IO is flushed

This looks more complicated than a single new state:

1) device is stopped and IO is flushed

Thanks

> 
> > 
> > > 
> > > > Making it clearer what does "complete" mean here might help.
> > > > 
> > > I think what we are trying to accomplish here, is avoiding, having
> > > non-standardised state in device (that can not be dropped) when
> > > migrating.
> > > 
> > > I'm still struggling with wrapping my mind around the problem. AFAIU
> > > migration and migratability is not really a feature of the virtio
> > > standard, but can be a feature of it's implementation
> > > (e.g. QEMU & KVM), where the standard does help a great deal by having
> > > certain aspects of the operation and interaction nailed down.
> > 
> > 
> > It looks like a must (at least from the level of semantics) for designing
> > software API for vDPA. Using a standard spec may help to avoid subtle
> > misunderstanding of different vendors.
> > 
> > Thanks
> > 
> > 
> > > 
> > > Regards,
> > > Halil
> > > 
> 
>