Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Max Gurtovoy <mgurtovoy@nvidia.com>]

On 06/03/2023 13:20, Stefan Hajnoczi wrote:
> On Mon, Mar 06, 2023 at 04:00:50PM +0800, Jason Wang wrote:
> > å 2023/3/6 08:03, Stefan Hajnoczi åé:
> > > On Sun, Mar 05, 2023 at 04:38:59AM -0500, Michael S. Tsirkin wrote:
> > > > On Fri, Mar 03, 2023 at 03:21:33PM -0500, Stefan Hajnoczi wrote:
> > > > > What happens if a command takes 1 second to complete, is the device
> > > > > allowed to process the next command from the virtqueue during this time,
> > > > > possibly completing it before the first command?
> > > > >
> > > > > This requires additional clarification in the spec because "they are
> > > > > processed by the device in the order in which they are queued" does not
> > > > > explain whether commands block the virtqueue (in order completion) or
> > > > > not (out of order completion).
> > > > Oh I begin to see. Hmm how does e.g. virtio scsi handle this?
> > > virtio-scsi, virtio-blk, and NVMe requests may complete out of order.
> > > Several may be processed by the device at the same time.
> > >
> > > They rely on multi-queue for abort operations:
> > >
> > > In virtio-scsi the abort requests (VIRTIO_SCSI_T_TMF_ABORT_TASK) are
> > > sent on the control virtqueue. The the request identifier namespace is
> > > shared across all virtqueues so it's possible to abort a request that
> > > was submitted to any command virtqueue.
> > >
> > > NVMe also follows the same design where abort commands are sent on the
> > > Admin Submission Queue instead of an I/O Submission Queue. It's possible
> > > to identify NVMe requests by <Submission Queue ID, Command Identifier>.
> > >
> > > virtio-blk doesn't support aborting requests.
> > >
> > > I think the logic behind this design is that if a queue gets stuck
> > > processing long-running requests, then the device should not be forced
> > > to perform lookahead in the queue to find abort commands. A separate
> > > control/admin queue is used for the abort requests.
> >
> >
> > Or device need mandate some kind of QOS here, e.g a request must be complete
> > in some time. Otherwise we don't have sufficient reliability for using it as
> > management task?
>
> Yes, if all commands can be executed in bounded time then a guarantee is
> possible.
>
> Here is an example where that's hard: imagine a virtio-blk device backed
> by network storage. When an admin queue command is used to delete a
> group member, any of the group member's in-flight I/O requests need to
> be aborted. If the network hangs while the group member is being
> deleted, then the device can't complete an orderly shutdown of I/O
> requests in a reasonable time.
>
> That example shows a basic group admin command that I think Michael is
> about to propose. We can't avoid this problem by not making it a group
> admin command - it needs to be a group admin command. So I think it's
> likely that there will be admin commands that take an unbounded amount
> of time to complete. One way to achieve what you mentioned is timeouts.

I think that you're getting into device specific implementation details 
and I'm not sure it's necessary.

I don't think we need to abort admin commands. Admin commands can be 
flushed/aborted during the device reset phase.
Only IO commands should have the possibility to being aborted as you 
mentioned in NVMe and SCSI (and potentially in virtio-blk).

For your example, stopping a member is possible even it there are some 
errors in the network. You can for example destroy all the connections 
to the remote target and complete all the BIOS with some error.

> Stefan