Re: [PATCH 2/5] Introduce VIRTIO_F_ADMIN_VQ_INDIRECT_DESC/VIRTIO_F_ADMIN_VQ_IN_ORDER

["Michael S. Tsirkin" <mst@redhat.com>]

On Tue, Jan 18, 2022 at 05:17:06PM +0000, Parav Pandit wrote:
> 
> 
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Tuesday, January 18, 2022 8:39 PM
> > 
> > On Tue, Jan 18, 2022 at 10:50:52AM +0000, Parav Pandit wrote:
> > >
> > >
> > > > From: Michael S. Tsirkin <mst@redhat.com>
> > > > Sent: Tuesday, January 18, 2022 4:09 PM
> > > >
> > > > On Tue, Jan 18, 2022 at 07:30:34AM +0000, Parav Pandit wrote:
> > > > > An unrelated command to AQ in Jason's proposal [1] is about " The
> > > > management driver MUST create a managed device by allocating".
> > > > > We see that creator of the subfunction is often not the only
> > > > > entity managing
> > > > it.
> > > > > They being same in new era finding less and less users.
> > > > > So this piece needs more discussion whenever we address that.
> > > > >
> > > > > [1]
> > > > > https://lists.oasis-open.org/archives/virtio-comment/202108/msg001
> > > > > 36.h
> > > > > tml
> > > >
> > > > This reminds me. How do AQ commands interact with VF lifecycle?
> > > VF device usage is controlled by the same system which is configuring the VF
> > via its parent PF device.
> > > So VF device shouldn't be in use. Any configuration change while VF device is
> > in use will result in failing the AQ command.
> > >
> > > > E.g. can one change number of vectors for an active VF?
> > > > Need to specify this.
> > > >
> > > > Also, I started worrying about compatibility here.
> > > > Let's say the msix capability in a VF specifies 16 vectors.
> > > > Can PF specify 32? If yes how will driver program them?
> > > Yes, PF can change to 32. When VF driver queries the PCI capability, it will
> > reflect 32 instead of 16.
> > > > Can PF specify 8? If yes how do we make sure driver does not attempt
> > > > to use 16? And what happens if it does?
> > > PF is programming the VF msix capability in the device. So virtio pci driver
> > operating the PCI VF device cannot access vectors beyond the max value
> > programmed by the PF driver.
> > 
> > Um. Interesting. This means that the msix capability of the VF changes?
> Yes.
> > Is that in fact spec compliant? Could some OSes cache the value of the
> > capability even if the device is not in active use? E.g. I can see how this might
> > happen in order to map the MSIX tables even before loading the driver.
> > 
> PCI subsystem can catch the value before the device driver can load.
> Generally a device support intx/msix or intx/msi. So PCI subsystem is not aware what will be used by its upper layer device drivers.
> So it usually differs such initialization to a later stage until it is actually used.
> 
> Whichever OS driver which implements msix configuration, will have to either not cache it or flush+ rebuild the cache.

Seems to contradict what the spec says (below).

> > The spec says:
> > 	Depending upon system software policy, system software, device driver
> > software, or each at
> > 	different times or environments may configure a functionâs MSI-X
> > capability and table
> > 	structures with suitable vectors.
> > 
> > So MSIX canfiguation might not be up to the driver.
> > 
> > We actually ask driver to read back any vector assigned to a VQ so it's possible
> > to fail vector assignment. Maybe that's better.
> > 
> Virtio driver should not incur any additional complexity in re-reading vector etc.

I think it does this already.

> All the msix config should happen much before drivers gets loaded for the VF.
> It is PCI layer of the HV to provide a stable device to virtio device driver which is not undergoing msix table changes, when virtio device driver is operating on it.


Problem is in the guest though. I'm not sure we can rely on this part
being part of the driver and not part of the OS.

-- 
MST