RE: [PATCH 2/5] Introduce VIRTIO_F_ADMIN_VQ_INDIRECT_DESC/VIRTIO_F_ADMIN_VQ_IN_ORDER

[Parav Pandit <parav@nvidia.com>]

> From: Jason Wang <jasowang@redhat.com>
> Sent: Wednesday, January 26, 2022 12:24 PM
> 
> > >
> > > 1) the vendor and transport that doesn't want to use admin virtqueue
> > It is not the choice of vendor to use admin virtqueue or not. It is the spec
> definition.
> 
> We are discussing the proposal which hasn't been a part of the spec right now.
> It doesn't mean we can't do better.
Sure. Itâs the proposal we are discussing, but I fail to understand why a vendor doesnât want to use a admin queue due to which it needs to be detached.

> >
> > Why a transport doesnât want to use admin queue?
> 
> The answer is simple and straightforward, each transport had already had its
> transport specific way to configure the device.
Each transport uses transport agnostic way to communicate the requests and response to the device by means of virtqueue descriptors.

And that is done through virtqueue.

A descriptor in a virtqueue represents wide range of things.
1. Sometime sending packet, 
2. sometimes configuring vlan,
3. sometime configuring adding mac to table.
4. sometime adding crypto session
5. and now sometime managing a managed VF or SF device

Item #5 is no different than #1 to #4.
Managed device configuration != "generic configure the device".

> 
> > I donât follow why virtio fs device wants to use something other than request
> queue to transport virtio_fs_req.
> 
> See my previous reply, I didn't mean we need to change any device specific
> operation like virtio_fs_req.
> 
You imply that virtio_fs_req must be carried on the request queue, but some other request like to use other than virtqueue even though its explained several times that why virtqueue is used.
Doesn't make sense at all. 

> What I meant is, let's take MSI as an example
> 
> 1) PCI has MSI table
> 2) MMIO doesn't support MSI
> 
> It looks to me you want to mandate admin virtqueue to MMIO in order to
> configure MSI?
> 
If you want to attribute it as "mandate" than yes, it is no different than,
virtio_fs_req is mandated on request vq.
virtio_crypto_op_ctrl_req ia mandated on control vq.

What prevents both above examples to used AQ in generic manner?

> >
> > > 2) a more simple interface for L1
> > I donât see virtqueue as complicated object which exists for 10+ years now
> and in use by 18 devices in L1 and also in L2.
> > And it is something to be used for multiple commands.
> 
> So again, there's misunderstanding. It's all about the "control plane"
> but not "dataplane". A simple example is to configure the vq address.
> 
> 1) For PCI it was done via common_cfg structure
> 2) For MMIO it was done via dedicated registers
> 3) For CCW, it was done via dedicated commands

We discussed already and you ignored the scale, on-die and other comments.
And low level self-device configuration is compared with managing device configuration.
This comparison simply isn't right.

data plane => vq
control plane => cq + transport specific cfg
mgmt plane => aq (this proposal)

mgmt plane != device on vq configuration.
Device on vq configuration doesn't block with other device VQ configuration.

> 
> And we know we need dedicated commands for the admin virtqueue. 
Not sure I follow you. AQ carries admin queue commands like any other queues.

> Since we
> had a transport specific interface, guest drivers can still configure the virtqueue
> address in L1 via the transport specific interface. So the hypervisor can hide the
> admin virtqueue. Let's say we introduce a new feature X for the admin
> virtqueue only. How can a guest driver to use that feature? 
As the patch-1, and patch-4 clearly shows the purpose of the AQ, AQ is of the virtio device located in the HV is not for the consumption in the guest driver.

> Do we want to hide
> or assign the admin virtqueue to L1?
A virtio device in guest may have its own admin queue. But again, until now there is no concrete example discussed that demands an admin queue in the guest driver.
IMS vector enable/disable was the closest one we discussed and we agreed that AQ cannot be used for that.

> 
> I understand that for things like provisioning which might not be needed for L1,
> but how about others?
Others?

> > >
> > Shahaf and I already explained in this thread that this capability doesnât scale
> to your question about "I do not understand scale ..".
> > So a device that wants to do above can simply do this with AQ with single VQ
> depth and still achieve the simplicity.
> 
> I thought it was for SF but not SR-IOV.
Ah, that was the misunderstanding you had. It applies to SF and SR-IOV both.

> 
> I think we all know SR-IOV doesn't scale in many ways and if I understand this
> series correctly, the main goal is not to address the scalability issues.
You are mixing many things here. Do not mix SR-IOV VF scaling with PF resource scaling.
Please keep it focused.

The goal as constantly repeated, is to have a interface between driver and device to carry multiple commands in scalable way without consuming on-chip resources.

> 
> It's good to consider the scalability but there could be cases that don't need to
> be scaled.
Sure, in that case AQ of single entry is just fine.

> 
> >
> > Michael also responded that device configuration will not end at msix
> configuration.
> > So adding more and more tiny capabilities for each configuration doesn't
> scale either.
> >
You need to ack above point otherwise we will keep discussing this even 2 weeks from now. :)

> > > >
> > > > Or you mean, a guest driver of VF or SF is configuring its IMS to
> > > > later
> > > consume for the VQ?
> > > > If its this, than I explained that admin queue is not the vehicle
> > > > to do so, and
> > > we discussed the other structure yday.
> > >
> > > Yes, I guess that's the nesting case I mentioned above.
> > >
> > > >
> > > > > 1) transport independent way: e.g admin virtqueue (which will be
> > > > > eventually became another transport)
> > > > >
> > > > IMS by guest driver cannot be configured by AQ.
> > >
> > > Yes, that's one point.
> > >
> > Ok. cool. We sync here too. :)
> 
> So I guess you know what I meant for the L1 interface?
With your last question, let me double check my understanding. :)

My understanding of L1 interface is virtio device in hypervisor. Is that right?