Re: [virtio-comment] [PATCH v1 1/8] admin: Add theory of operation for device migration

["Michael S. Tsirkin" <mst@redhat.com>]

On Thu, Oct 19, 2023 at 09:20:22AM +0000, Parav Pandit wrote:
> 
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Thursday, October 19, 2023 2:41 PM
> > 
> > On Thu, Oct 19, 2023 at 08:58:10AM +0000, Parav Pandit wrote:
> > > It cannot be in the PCI 4K config space for sure.
> > > It must reside in the virtio config space.
> > 
> > Why? 
> Because pci spec has clearly called out to not place any device specific things in there.
> 
> Citation in pci spec
> " It is strongly recommended that PCI Express devices place no registers in Configuration Space other than those in
> headers or Capability structures architected by applicable PCI specifications."

But of course, we'd place them inside a vendor specific capability.
We just need a version of virtio_pci_cap64 that's suitable for express.


> > My concern with virtio config space would be that it's not orthogonal to
> > other things in the config space. E.g. you need to look at feature bits to discover
> > presence. How does this work while you are documenting that device can
> > undergo reset at any time?
> This is why such solution cannot work for passthrough. It can only fulfil #2 based approach.
> 
> > With a capability you can discover it without poking at features.
> > 
> It is discouraged by pci spec.
> Pci caps for mostly doing very small init time sort of config.
> Not to run frequent commands.

Interesting. where in the spec exactly?

> > > I am sure that this is used for passthrough mode of #1.
> > > So, can you please confirm to write this up for mode #2 only?
> > 
> > To me it sounds like a generally useful capability that could be used as basis e.g.
> > for admin command transport.
> > 
> Unfortunately, it cannot be pci capability. 
> It needs to stay in virtio area and only fulfill use case of #2.
> 
> > > > > A variation of that for the member device, there is owner device,
> > > > > hence
> > > > admin command on the AQ can be used.
> > > > >
> > > > > If we can converge on common virtio interface between #1 and #2, great.
> > > > > If we cannot be due to technical issues, we shouldn't step on each
> > > > > other's
> > > > toes, instead build the two interfaces for two different use cases
> > > > overcoming its own technical challenges.
> > > > >
> > > > > And when in future, someone want to implement different kind of
> > > > > bisections,
> > > > they can propose the extensions.
> > > >
> > > > Not good at all, this means the interface is very narrow.
> > > > Your "propose an extension" just doesn't work practically.
> > > > It takes years for things to be widely deployed in the field, by the
> > > > time they are there are more use-cases.
> > >
> > > We usually see it getting deployed in < 1 year time with new spec
> > advancement pace for many features.
> > > Building something for unreasonable amount of time without use case results
> > in missing the immediate deployments that happens in 2024 to 2027 of 1.4 spec
> > time frame.
> > >
> > > > We need something universal and admin commands were supposed to be
> > > > just this.
> > > I don't see a universal solution for all problems for above #1 and #2.
> > >
> > > Solving above #2 will cover large part of deployments that users are doing.
> > 
> > OK. But additionally, if an interface can cover a couple of use-cases we can be
> > reasonably sure it's going to cover more going forward.
> May be.

Yes hard to be sure. But if it can't then that's a good sign it's
problematic.

-- 
MST