Re: [virtio-comment] [PATCH RFC] virtio: introduce VIRTIO_F_DEVICE_STOP

["Michael S. Tsirkin" <mst@redhat.com>]

On Mon, Dec 28, 2020 at 03:01:57PM +0800, Jason Wang wrote:
> 
> On 2020/12/28 äå2:21, Halil Pasic wrote:
> > On Sun, 27 Dec 2020 05:00:05 -0500
> > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > 
> > > On Fri, Dec 25, 2020 at 08:38:35AM +0100, Halil Pasic wrote:
> > > > When driver is trying to set DEVICE_STOPPED, the device MUST not
> > > > process new avail requests and MUST complete all requests that is
> > > > currently processing before setting DEVICE_STOPPED.
> > > ...
> > > 
> > > > Since we have a synchronous API setting DEVICE_STOPPED would also have to
> > > > block until all in-flight requests are completed.
> > > Judging from the surrounding discussion,
> > > when you say complete you seem to mean "use", and
> > My intention was merely to paraphrase Jason's proposal which says:
> > 
> > +When driver is trying to set DEVICE_STOPPED, the device MUST not
> > +process new avail requests and MUST complete all requests that is
> > +currently processing before setting DEVICE_STOPPED.
> > 
> > > I'm not sure how you define in flight, but
> > > it seems there could be many of these (up to a full queue?)
> > In the follow-on discussion 'in-flight' emerged as an alternative to
> > 'all requests that is currently processing' form the proposed text.
> > 
> > A large part of the discussion is, IMHO, about finding precise
> > definitions, for what the quoted paragraph is trying to express.
> > 
> > > so waiting for all available buffers to be
> > > used might indeed require an asynchronous interface,
> > > which gets complex very quickly.
> 
> 
> Some part of the virtio has enforced an asynchronous interface during reset:
> 
> For MMIO the spec said:
> 
> """
> 
> To stop using the queue the driver MUST write zero (0x0) to this QueueReady
> and MUST read the value back to ensure synchronization.
> 
> """
> 
> For PCI it said:
> 
> """
> 
> After writing 0 to device_status, the driver MUST wait for a read of
> device_status to return 0 before reinitializing the device.
> 
> """

Absolutely. It's ok for simple things.
However doing anything major like waiting for IO
in a tight loop is a problem.



> 
> > > 
> > > However wouldn't it be possible for device to just cancel
> > > processing available buffers even if it started processing
> > > them? Some buffers could be in indeterminate state
> > > (e.g. we might not have a way to know how much data did
> > > device have time to write into a buffer).
> > > 
> > Maybe Jason can answer this one.
> 
> 
> For networking device, it should be possible (packet could be dropped or
> duplicated). But I'm not sure it works for other device. E.g for the block
> devices that needs to communicate with a remote backend. Waat's more
> important, my understadning is that the intermediate state is something that
> we need to avoid (hard to be migrated anyhow).

The difficulty is on the device side though, so why not say: if it wants to
flush IO that's up to it.

> 
> > 
> > > Making it clearer what does "complete" mean here might help.
> > > 
> > I think what we are trying to accomplish here, is avoiding, having
> > non-standardised state in device (that can not be dropped) when
> > migrating.
> > 
> > I'm still struggling with wrapping my mind around the problem. AFAIU
> > migration and migratability is not really a feature of the virtio
> > standard, but can be a feature of it's implementation
> > (e.g. QEMU & KVM), where the standard does help a great deal by having
> > certain aspects of the operation and interaction nailed down.
> 
> 
> It looks like a must (at least from the level of semantics) for designing
> software API for vDPA. Using a standard spec may help to avoid subtle
> misunderstanding of different vendors.
> 
> Thanks
> 
> 
> > 
> > Regards,
> > Halil
> >