RE: [virtio-comment] [PATCH v1 1/8] admin: Add theory of operation for device migration

[Parav Pandit <parav@nvidia.com>]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Thursday, October 19, 2023 3:20 PM
> 
> On Thu, Oct 19, 2023 at 09:39:48AM +0000, Parav Pandit wrote:
> >
> > > From: Michael S. Tsirkin <mst@redhat.com>
> > > Sent: Thursday, October 19, 2023 2:57 PM
> > >
> > > On Thu, Oct 19, 2023 at 09:20:22AM +0000, Parav Pandit wrote:
> > > >
> > > > > From: Michael S. Tsirkin <mst@redhat.com>
> > > > > Sent: Thursday, October 19, 2023 2:41 PM
> > > > >
> > > > > On Thu, Oct 19, 2023 at 08:58:10AM +0000, Parav Pandit wrote:
> > > > > > It cannot be in the PCI 4K config space for sure.
> > > > > > It must reside in the virtio config space.
> > > > >
> > > > > Why?
> > > > Because pci spec has clearly called out to not place any device
> > > > specific things
> > > in there.
> > > >
> > > > Citation in pci spec
> > > > " It is strongly recommended that PCI Express devices place no
> > > > registers in Configuration Space other than those in headers or
> > > > Capability
> > > structures architected by applicable PCI specifications."
> > >
> > > But of course, we'd place them inside a vendor specific capability.
> > > We just need a version of virtio_pci_cap64 that's suitable for express.
> > >
> > It is one and the same thing, wrapping dma commands using capability, is
> same as open coding them.
> > Read only cap should say where admin command section is located, which
> must be done when the DRIVER_OK is done.
> 
> This seems very different from your current commands.
> There's less value in reusing commands if semantics are subtly different ..
> 
I mean read only cap points to a location in BAR.
And mediation software who fully owns the member device, issues side command there after DRIVER_OK.

> > For sure, we wont use this for passthrough member devices.
> >
> > So before discussing where to place them, it is fundamental to agree its use
> case which is #2.
> 
> Again components need to be versatile, not just focused on one use-case.
> 
Don't see how one can make a tire in a truck and car both. :)
We are trying, so lets see, may be we can.

> 
> > >
> > > > > My concern with virtio config space would be that it's not
> > > > > orthogonal to other things in the config space. E.g. you need to
> > > > > look at feature bits to discover presence. How does this work
> > > > > while you are documenting that device can undergo reset at any time?
> > > > This is why such solution cannot work for passthrough. It can only
> > > > fulfil #2
> > > based approach.
> > > >
> > > > > With a capability you can discover it without poking at features.
> > > > >
> > > > It is discouraged by pci spec.
> > > > Pci caps for mostly doing very small init time sort of config.
> > > > Not to run frequent commands.
> > >
> > > Interesting. where in the spec exactly?
> > >
> > Same above section, section 7.2.2.2 implementation notes.
> 
> I see. So the valid use case is access before memory is enabled.
> But in fact, e.g. FLR actually disables memory does it not?
It does. This is why all this VF resident registers only work for non-passthrough area.

> So if we want same behaviour as you are proposing here then these need to
> work with memory disabled yes?
> 
> BTW they also recommend against read side effects which virtio violates :(
> 
Yes.
This is why staying away from these caps which is not the vehicle for admin commands.

> 
> > > > > > I am sure that this is used for passthrough mode of #1.
> > > > > > So, can you please confirm to write this up for mode #2 only?
> > > > >
> > > > > To me it sounds like a generally useful capability that could be
> > > > > used as basis
> > > e.g.
> > > > > for admin command transport.
> > > > >
> > > > Unfortunately, it cannot be pci capability.
> > > > It needs to stay in virtio area and only fulfill use case of #2.
> > > >
> > > > > > > > A variation of that for the member device, there is owner
> > > > > > > > device, hence
> > > > > > > admin command on the AQ can be used.
> > > > > > > >
> > > > > > > > If we can converge on common virtio interface between #1
> > > > > > > > and #2,
> > > great.
> > > > > > > > If we cannot be due to technical issues, we shouldn't step
> > > > > > > > on each other's
> > > > > > > toes, instead build the two interfaces for two different use
> > > > > > > cases overcoming its own technical challenges.
> > > > > > > >
> > > > > > > > And when in future, someone want to implement different
> > > > > > > > kind of bisections,
> > > > > > > they can propose the extensions.
> > > > > > >
> > > > > > > Not good at all, this means the interface is very narrow.
> > > > > > > Your "propose an extension" just doesn't work practically.
> > > > > > > It takes years for things to be widely deployed in the
> > > > > > > field, by the time they are there are more use-cases.
> > > > > >
> > > > > > We usually see it getting deployed in < 1 year time with new
> > > > > > spec
> > > > > advancement pace for many features.
> > > > > > Building something for unreasonable amount of time without use
> > > > > > case results
> > > > > in missing the immediate deployments that happens in 2024 to
> > > > > 2027 of
> > > > > 1.4 spec time frame.
> > > > > >
> > > > > > > We need something universal and admin commands were supposed
> > > > > > > to be just this.
> > > > > > I don't see a universal solution for all problems for above #1 and #2.
> > > > > >
> > > > > > Solving above #2 will cover large part of deployments that users are
> doing.
> > > > >
> > > > > OK. But additionally, if an interface can cover a couple of
> > > > > use-cases we can be reasonably sure it's going to cover more going
> forward.
> > > > May be.
> > >
> > > Yes hard to be sure. But if it can't then that's a good sign it's problematic.
> > I don't see it problematic at all.
> > If we write the virtio specification for device migration in 1.4-time frame, I
> am 100% sure that it will be deployed before spec release.
> > Other uses should be able to extend it as they evolve and explain why the
> current one does not fit them.
> 
> My experience tells me this results in a huge mess of incompatible interfaces,
> the further you go down this road the harder it becomes to add new things as
> they conflict with old ones.
Too abstract for me.
Will focus on the above technical parts that is key to draft a common interface.