Re: [virtio-comment] [PATCH v1 1/8] admin: Add theory of operation for device migration

["Michael S. Tsirkin" <mst@redhat.com>]

On Wed, Nov 15, 2023 at 05:39:43PM +0000, Parav Pandit wrote:
> > >
> > > Additionally, if hypervisor has put the trap on virtio config, and
> > > because the memory device already has the interface for virtio config,
> > >
> > > Hypervisor can directly write/read from the virtual config to the member's
> > config space, without going through the device context, right?
> > 
> > If it can do it or it can choose to not. I don't see how it is related to the
> > discussion here.
> >
> It is. I donât see a point of hypervisor not using the native interface provided by the member device.

So for example, it seems reasonable to a member supporting both
existing pci register interface for compatibility and the future
DMA based one for scale. In such a case, it seems possible that
DMA will expose more features than pci. And then a hypervisor
might decide to use that in preference to pci registers.



>  > >
> > > >
> > > > >
> > > > > >
> > > > > > > it is not good idea to overload management commands with
> > > > > > > actual run time
> > > > > > guest commands.
> > > > > > > The device context read writes are largely for incremental updates.
> > > > > >
> > > > > > It doesn't matter if it is incremental or not but
> > > > > >
> > > > > It does because you want different functionality only for purpose
> > > > > of backward
> > > > compatibility.
> > > > > That also if the device does not offer them as portion of MMIO BAR.
> > > >
> > > > I don't see how it is related to the "incremental part".
> > > >
> > > > >
> > > > > > 1) the function is there
> > > > > > 2) hypervisor can use that function if they want and virtio
> > > > > > (spec) can't forbid that
> > > > > >
> > > > > It is not about forbidding or supporting.
> > > > > Its about what functionality to use for management plane and guest
> > plane.
> > > > > Both have different needs.
> > > >
> > > > People can have different views, there's nothing we can prevent a
> > > > hypervisor from using it as a transport as far as I can see.
> > > For device context write command, it can be used (or probably abused) to do
> > write but I fail to see why to use it.
> > 
> > The function is there, you can't prevent people from doing that.
> >
> One can always mess up itself. :)
> It is not prevented. It is just not right way to use the interface.
>  
> > > Because member device already has the interface to do config read/write and
> > it is accessible to the hypervisor.
> > 
> > Well, it looks self-contradictory again. Are you saying another set of commands
> > that is similar to device context is needed for non-PCI transport?
> >
> All these non pci transport discussion is just meaning less.
> Let MMIO bring the concept of member device at that point something make sense to discuss.
> PCI SIOV is also the PCI device at the end.
>  
> > >
> > > The read as_is using device context cannot be done because the caller is not
> > explicitly asking what to read.
> > > And the interface does not have it, because member device has it.
> > >
> > > So lets find the need if incremental bit is needed in the device_Context read
> > command or not or a bits to ask explicitly what to read optionally.
> > >
> > > >
> > > > >
> > > > > > >
> > > > > > > For VF driver it has own direct channel via its own BAR to
> > > > > > > talk to the
> > > > device.
> > > > > > So no need to transport via PF.
> > > > > > > For SIOV for backward compat vPCI composition, it may be needed.
> > > > > > > Hard to say, if that can be memory mapped as well on the BAR of the
> > PF.
> > > > > > > We have seen one device supporting it outside of the virtio.
> > > > > > > For scale anyway, one needs to use the device own cvq for
> > > > > > > complex
> > > > > > configuration.
> > > > > >
> > > > > > That's the idea but I meant your current proposal overlaps those
> > functions.
> > > > > >
> > > > > Not really. One can have simple virtio config space access
> > > > > read/write
> > > > functionality, in addition to what is done here.
> > > > > And that is still fine. One is doing proxying for guest.
> > > > > Management plane is doing more than just register proxy.
> > > >
> > > > See above, let's figure out whether it is possible as a transport first then.
> > > >
> > > Right. lets figure out.
> > >
> > > I would still promote to not mix management command with transport
> > command.
> > 
> > It's not a mixing, it's just because they are functional equivalents.
> > 
> It is not.
> I clarified the fundamental difference between the two.
> One is explicit read and write.
> Other is, return read data on change.
> For write, it is explicit set and it does not take effect until the mode is changed back to active.
> 
> > > Commands are cheap in nature. For transport if needed, they can be explicit
> > commands.
> > 
> > It will be a partial duplication of what is being proposed here.
> 
> There is always some overlap between management plane (hypervisor set/get) and control plane (guest driver get/set).
> > 
> > Thanks
> > 
> > 
> > 
> > >
> > > > >
> > > > > > >
> > > > > > > > >
> > > > > > > > > > > If for that is some admin commands are missing, may be
> > > > > > > > > > > one can add
> > > > > > > > them.
> > > > > > > > > >
> > > > > > > > > > I would then build the device context commands on top of
> > > > > > > > > > the transport commands/q, then it would be complete.
> > > > > > > > > >
> > > > > > > > > > > No need to step on toes of use cases as they are different...
> > > > > > > > > > >
> > > > > > > > > > > > I've shown you that
> > > > > > > > > > > >
> > > > > > > > > > > > 1) you can't easily say you can pass through all the
> > > > > > > > > > > > virtio facilities
> > > > > > > > > > > > 2) how ambiguous for terminology like "passthrough"
> > > > > > > > > > > >
> > > > > > > > > > > It is not, it is well defined in v3, v2.
> > > > > > > > > > > One can continue to argue and keep defining the
> > > > > > > > > > > variant and still call it data
> > > > > > > > > > path acceleration and then claim it as passthrough ...
> > > > > > > > > > > But I won't debate this anymore as its just
> > > > > > > > > > > non-technical aspects of least
> > > > > > > > > > interest.
> > > > > > > > > >
> > > > > > > > > > You use this terminology in the spec which is all about
> > > > > > > > > > technical, and you think how to define it is a matter of
> > > > > > > > > > non-technical. This is self-contradictory. If you fail,
> > > > > > > > > > it probably means it's
> > > > > > ambiguous.
> > > > > > > > > > Let's don't use that terminology.
> > > > > > > > > >
> > > > > > > > > What it means is described in theory of operation.
> > > > > > > > >
> > > > > > > > > > > We have technical tasks and more improved specs to
> > > > > > > > > > > update going
> > > > > > > > forward.
> > > > > > > > > >
> > > > > > > > > > It's a burden to do the synchronization.
> > > > > > > > > We have discussed this.
> > > > > > > > > In current proposed the member device is not bifurcated,
> > > > > > > >
> > > > > > > > It is. Part of the functions were carried via the PCI
> > > > > > > > interface, some are carried via owner. You end up with two
> > > > > > > > drivers to drive the
> > > > > > devices.
> > > > > > > >
> > > > > > > Nop.
> > > > > > > All admin work of device migration is carried out via the owner
> > device.
> > > > > > > All guest triggered work is carried out using VF itself.
> > > > > >
> > > > > > Guests don't (or can't) care about how the hypervisor is structured.
> > > > > For passthrough mode, it just cannot be structured inside the VF.
> > > >
> > > > Well, again, we are talking about different things.
> > > >
> > > > >
> > > > > > So we're discussing the view of device, member devices needs to
> > > > > > server for
> > > > > >
> > > > > > 1) request from the transport (it's guest in your context)
> > > > > > 2) request from the owner
> > > > >
> > > > > Doing #2 of the owner on the member device functionality do not
> > > > > work when
> > > > hypervisor do not have access to the member device.
> > > >
> > > > I don't get here, isn't 2) just what we invent for admin commands?
> > > > Driver sends commands to the owner, owner forward those requests to
> > > > the member?
> > > I am most with the term "driver" without notion of guest/hypervisor prefix.
> > >
> > > In one model,
> > > Member device does everything through its native interface = virtio config
> > and device space, cvq, data vqs etc.
> > > Here member device do not forward anything to its owner.
> > >
> > > The live migration hypervisor driver who has the knowledge of live migration
> > flow, accesses the owner device and get the side band member's information to
> > control it.
> > > So member driver do not forward anything here to owner driver.
> > >
>