Re: [virtio-comment] [PATCH v1 1/8] admin: Add theory of operation for device migration

["Michael S. Tsirkin" <mst@redhat.com>]

On Thu, Oct 19, 2023 at 05:26:54AM -0400, Michael S. Tsirkin wrote:
> On Thu, Oct 19, 2023 at 09:20:22AM +0000, Parav Pandit wrote:
> > 
> > > From: Michael S. Tsirkin <mst@redhat.com>
> > > Sent: Thursday, October 19, 2023 2:41 PM
> > > 
> > > On Thu, Oct 19, 2023 at 08:58:10AM +0000, Parav Pandit wrote:
> > > > It cannot be in the PCI 4K config space for sure.
> > > > It must reside in the virtio config space.
> > > 
> > > Why? 
> > Because pci spec has clearly called out to not place any device specific things in there.
> > 
> > Citation in pci spec
> > " It is strongly recommended that PCI Express devices place no registers in Configuration Space other than those in
> > headers or Capability structures architected by applicable PCI specifications."
> 
> But of course, we'd place them inside a vendor specific capability.
> We just need a version of virtio_pci_cap64 that's suitable for express.
> 
> 
> > > My concern with virtio config space would be that it's not orthogonal to
> > > other things in the config space. E.g. you need to look at feature bits to discover
> > > presence. How does this work while you are documenting that device can
> > > undergo reset at any time?
> > This is why such solution cannot work for passthrough. It can only fulfil #2 based approach.
> > 
> > > With a capability you can discover it without poking at features.
> > > 
> > It is discouraged by pci spec.
> > Pci caps for mostly doing very small init time sort of config.
> > Not to run frequent commands.
> 
> Interesting. where in the spec exactly?

And the reason I ask is because I'd like to understand the exact
limitation.

In any case, we should still maybe look for ways to separate it from config.
Maybe a capability points at a BAR and that is where we
have this stuff?

 
> > > > I am sure that this is used for passthrough mode of #1.
> > > > So, can you please confirm to write this up for mode #2 only?
> > > 
> > > To me it sounds like a generally useful capability that could be used as basis e.g.
> > > for admin command transport.
> > > 
> > Unfortunately, it cannot be pci capability. 
> > It needs to stay in virtio area and only fulfill use case of #2.


> > > > > > A variation of that for the member device, there is owner device,
> > > > > > hence
> > > > > admin command on the AQ can be used.
> > > > > >
> > > > > > If we can converge on common virtio interface between #1 and #2, great.
> > > > > > If we cannot be due to technical issues, we shouldn't step on each
> > > > > > other's
> > > > > toes, instead build the two interfaces for two different use cases
> > > > > overcoming its own technical challenges.
> > > > > >
> > > > > > And when in future, someone want to implement different kind of
> > > > > > bisections,
> > > > > they can propose the extensions.
> > > > >
> > > > > Not good at all, this means the interface is very narrow.
> > > > > Your "propose an extension" just doesn't work practically.
> > > > > It takes years for things to be widely deployed in the field, by the
> > > > > time they are there are more use-cases.
> > > >
> > > > We usually see it getting deployed in < 1 year time with new spec
> > > advancement pace for many features.
> > > > Building something for unreasonable amount of time without use case results
> > > in missing the immediate deployments that happens in 2024 to 2027 of 1.4 spec
> > > time frame.
> > > >
> > > > > We need something universal and admin commands were supposed to be
> > > > > just this.
> > > > I don't see a universal solution for all problems for above #1 and #2.
> > > >
> > > > Solving above #2 will cover large part of deployments that users are doing.
> > > 
> > > OK. But additionally, if an interface can cover a couple of use-cases we can be
> > > reasonably sure it's going to cover more going forward.
> > May be.
> 
> Yes hard to be sure. But if it can't then that's a good sign it's
> problematic.
> 
> -- 
> MST