RE: [PATCH 2/5] Introduce VIRTIO_F_ADMIN_VQ_INDIRECT_DESC/VIRTIO_F_ADMIN_VQ_IN_ORDER

[Parav Pandit <parav@nvidia.com>]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Thursday, January 27, 2022 6:36 PM
> 
> On Thu, Jan 27, 2022 at 03:49:58AM +0000, Parav Pandit wrote:
> >
> > > From: Michael S. Tsirkin <mst@redhat.com>
> > > Sent: Wednesday, January 26, 2022 7:42 PM
> > >
> > > On Wed, Jan 26, 2022 at 01:29:27PM +0000, Parav Pandit wrote:
> > > >
> > > > > From: Michael S. Tsirkin <mst@redhat.com>
> > > > > Sent: Tuesday, January 25, 2022 5:39 PM
> > > >
> > > > > >
> > > > > > So you do agree that managing a managed
> > > > > > (create/destroy/setup/etc...) will be done using the AQ of the
> > > > > > managing
> > > device ?
> > > > >
> > > > > I think Jason asked that the management commands are split from
> > > > > the queue itself, such that they can be implemented in more ways
> > > > > down the
> > > road.
> > > >
> > > > Admin commands involved DMA. And also multiple commands can be
> > > enqueued to the device.
> > > > I don't see any other construct in the spec other than vq that
> > > > enables driver
> > > and device to achieve both.
> > > > Hence split the queue from command is very weird for spec to do.
> > >
> > >  VIRTIO_PCI_CAP_SHARED_MEMORY_CFG can do this.
> > >
> > Device needs to always have shared memory region always in available
> range for driver to read.
> > Driver still needs to copy data from shared region to own memory.
> > How DMA Can be done in the host driver PA using shared memory region?
> 
> It can't but you don't need DMA if device is writing its own memory.
And sometimes host also needs to write. So DMA is required.

> 
> > In future when SIOV occurs, each new SIOV SF/ADI will require new
> > window in the shared regions exposed statically during PCI discovery
> > time.
> 
> Donnu if it's statically. But we are talking about AQ thing, that is just per group
> not per SF, so might not be too bad.
I must repeat again that it requires memory to be always available at scale.
For example if a virtio device state requires 1Kbytes of metadata, you are asking device to implement 1Mbytes of memory to be accessible in PCI read latency time window.
This is super waste of device resources for querying objects used only during mgmt. time.

If you were thinking to use this shared memory regions as small region and using it for command queuing interface, than probably yes, but than it starts looking like spec's well defined VQ object.
And at that point reinventing VQ of shared memory just doesnât make much sense.

> > Since we agree for this, I considered AQ vs no_AQ as closed topic?
> 
> It's always hard to answer questions like this when all I have to go on is a vague
> idea of what you mean. 
Which area is still vague? Can you please be specific, so that it can be covered in v3?
We answered all or almost the queries.
We addressed the comments you requested in V2.
Did you get a chance to review v2?

Jason's comment of splitting admin commands from admin queue is major item to fix in v3.
Documenting section differently, some in transport specific area was another fix to happen in v3.

In v3, if we are again going to discuss AQ vs config mmio region vs shared region, than we should discuss now before posting v3.
So please let me know if this area is vague or something else.