RE: [virtio-comment] [PATCH v1 1/8] admin: Add theory of operation for device migration

[Parav Pandit <parav@nvidia.com>]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Thursday, October 19, 2023 2:57 PM
> 
> On Thu, Oct 19, 2023 at 09:20:22AM +0000, Parav Pandit wrote:
> >
> > > From: Michael S. Tsirkin <mst@redhat.com>
> > > Sent: Thursday, October 19, 2023 2:41 PM
> > >
> > > On Thu, Oct 19, 2023 at 08:58:10AM +0000, Parav Pandit wrote:
> > > > It cannot be in the PCI 4K config space for sure.
> > > > It must reside in the virtio config space.
> > >
> > > Why?
> > Because pci spec has clearly called out to not place any device specific things
> in there.
> >
> > Citation in pci spec
> > " It is strongly recommended that PCI Express devices place no
> > registers in Configuration Space other than those in headers or Capability
> structures architected by applicable PCI specifications."
> 
> But of course, we'd place them inside a vendor specific capability.
> We just need a version of virtio_pci_cap64 that's suitable for express.
> 
It is one and the same thing, wrapping dma commands using capability, is same as open coding them.
Read only cap should say where admin command section is located, which must be done when the DRIVER_OK is done.

For sure, we wont use this for passthrough member devices.

So before discussing where to place them, it is fundamental to agree its use case which is #2.

> 
> > > My concern with virtio config space would be that it's not
> > > orthogonal to other things in the config space. E.g. you need to
> > > look at feature bits to discover presence. How does this work while
> > > you are documenting that device can undergo reset at any time?
> > This is why such solution cannot work for passthrough. It can only fulfil #2
> based approach.
> >
> > > With a capability you can discover it without poking at features.
> > >
> > It is discouraged by pci spec.
> > Pci caps for mostly doing very small init time sort of config.
> > Not to run frequent commands.
> 
> Interesting. where in the spec exactly?
> 
Same above section, section 7.2.2.2 implementation notes.

> > > > I am sure that this is used for passthrough mode of #1.
> > > > So, can you please confirm to write this up for mode #2 only?
> > >
> > > To me it sounds like a generally useful capability that could be used as basis
> e.g.
> > > for admin command transport.
> > >
> > Unfortunately, it cannot be pci capability.
> > It needs to stay in virtio area and only fulfill use case of #2.
> >
> > > > > > A variation of that for the member device, there is owner
> > > > > > device, hence
> > > > > admin command on the AQ can be used.
> > > > > >
> > > > > > If we can converge on common virtio interface between #1 and #2,
> great.
> > > > > > If we cannot be due to technical issues, we shouldn't step on
> > > > > > each other's
> > > > > toes, instead build the two interfaces for two different use
> > > > > cases overcoming its own technical challenges.
> > > > > >
> > > > > > And when in future, someone want to implement different kind
> > > > > > of bisections,
> > > > > they can propose the extensions.
> > > > >
> > > > > Not good at all, this means the interface is very narrow.
> > > > > Your "propose an extension" just doesn't work practically.
> > > > > It takes years for things to be widely deployed in the field, by
> > > > > the time they are there are more use-cases.
> > > >
> > > > We usually see it getting deployed in < 1 year time with new spec
> > > advancement pace for many features.
> > > > Building something for unreasonable amount of time without use
> > > > case results
> > > in missing the immediate deployments that happens in 2024 to 2027 of
> > > 1.4 spec time frame.
> > > >
> > > > > We need something universal and admin commands were supposed to
> > > > > be just this.
> > > > I don't see a universal solution for all problems for above #1 and #2.
> > > >
> > > > Solving above #2 will cover large part of deployments that users are doing.
> > >
> > > OK. But additionally, if an interface can cover a couple of
> > > use-cases we can be reasonably sure it's going to cover more going forward.
> > May be.
> 
> Yes hard to be sure. But if it can't then that's a good sign it's problematic.
I don't see it problematic at all.
If we write the virtio specification for device migration in 1.4-time frame, I am 100% sure that it will be deployed before spec release.
Other uses should be able to extend it as they evolve and explain why the current one does not fit them.