Re: [virtio-comment] Re: [PATCH v1 3/8] device-context: Define the device context fields for device migration

["Michael S. Tsirkin" <mst@redhat.com>]

On Fri, Oct 20, 2023 at 07:11:49PM +0800, Zhu, Lingshan wrote:
> 
> 
> On 10/20/2023 5:41 PM, Michael S. Tsirkin wrote:
> > On Fri, Oct 20, 2023 at 05:31:01PM +0800, Zhu, Lingshan wrote:
> > > 
> > > On 10/19/2023 6:33 PM, Parav Pandit wrote:
> > > > > From: Zhu, Lingshan <lingshan.zhu@intel.com>
> > > > > Sent: Thursday, October 19, 2023 2:48 PM
> > > > > 
> > > > > On 10/19/2023 5:14 PM, Michael S. Tsirkin wrote:
> > > > > > On Thu, Oct 19, 2023 at 09:13:16AM +0000, Parav Pandit wrote:
> > > > > > > > Oh, really? Quite interesting, do you want to move all config space
> > > > > > > > fields in VF to admin vq? Have a plan?
> > > > > > > Not in my plan for spec 1.4 time frame.
> > > > > > > I do not want to divert the discussion, would like to focus on device
> > > > > migration phases.
> > > > > > > Lets please discuss in some other dedicated thread.
> > > > > > Possibly, if there's a way to send admin commands to vf itself then
> > > > > > Lingshan will be happy?
> > > > > still need to prove why admin commands are better than registers.
> > > > Virtio spec development is not proof based approach. Please stop asking for it.
> > > > 
> > > > I tried my best to have technical answer in [1].
> > > > I explained that registers simply do not work for passthrough mode
> > > > (if this is what you are asking when you are asking prove its better).
> > > > They can work for non_passthrough mediated mode.
> > > > 
> > > > A member device may do admin commands using registers. Michael and I are discussing presently in the same thread.
> > > > 
> > > > Since there are multiple things to be done for device migration, dedicated register set for each functionality do not scale well, hard to maintain and extend.
> > > > A register holding a command content make sense.
> > > > 
> > > > Now, with that, if this can be useful only for non_passthrough, I made humble request to transport them using AQ, this way, you get all benefits of AQ.
> > > > And trying to understand, why AQ cannot possible or inferior?
> > > > 
> > > > If you have commands like suspend/resume device, register or queue transport simply donât work, because it's wrong to bifurcate the device with such weird API.
> > > > If you want to biferacate for mediation software, it probably makes sense to operate at each VQ level, config space level. Such are very different commands than passthrough.
> > > > I think vdpa has demonstrated that very well on how to do specific work for specific device type. So some of those work can be done using AQ.
> > > > 
> > > > [1] 870ace02-f99c-4582-932f-bd103362dae9@intel.com/T/#m37743aa924536d0256d6b3b8e83a11c750f28794">https://lore.kernel.org/virtio-comment/870ace02-f99c-4582-932f-bd103362dae9@intel.com/T/#m37743aa924536d0256d6b3b8e83a11c750f28794
> > > We have been through your statement for many times.
> > > This is not about how many times you repeated, if you think this is true,
> > > you need to prove that with solid evidence.
> > > 
> > > 
> > > For pass-through, I still recommend you to take a reference of current
> > > virito-pci implementation, it works for pass-through, right?
> > Current migration implementation in e.g. QEMU? It does but it
> > traps data path accesses. That, I think we can agree,
> > should not be the only option to migrate.
> OK, I am glad we agree that config space work for pass-through,
> hope we don't need to discuss this anymore.
> > 
> > > For scale, I already told you for many times that they are per-device
> > > facilities. How can a per-device facility not scale?
> > > vDPA works fine on config space.
> > > 
> > > So, if you still insist admin vq is better than config space like in other
> > > thread you have concluded, you may imply that config space interfaces should
> > > be re-factored to admin vq.
> > There are good arguments that yes, virtio needs a transport for config space
> > that is DMA based as opposed to memory mapped based.  This is one of the
> > things all vendors seem to prefer in IDPF so virtio should have the option.
> Do you really want to refactor virtio-pci common config fields to PF's admin
> vq?
> E.g, do you really want to move queue_enable in virtio-pci common config
> fields to PF's admin vq?

No, I think we need a transport with as small # of memory mapped registers as
possible that passes admin commands through the VF itself.

> Config space is control path, DMA is data-path, let's better not mix them,
> we never expect to use config space to transfer data.
> 
> So we need DMA to transfer data, for example I take advantages of device DMA
> to logging dirty pages, This also applies to in-flight descriptors.

As long as you do, I personally see little benefit to retrieve parts of
state with memory mapped accesses.

> And we are implementing virito live migration, not only for PCI.
> 
> So both me and Jason keep repeating: We are implementing basic facilities,
> and the implementation is transport specific.

But the register based facilities you proposed are extremely limited and
seem to only work for migration. For example, it seems mostly useless for
debugging because retrieving state is rather complex and would
interfere with normal working of the device.


> We have proposed to build admin vq based on our register solution, this can
> somehow even help tp resolve the nested issue.
> 
> But I see the proposed has been rejected.
> 
> I still believe the goal is to build a best spec, not "just can work" with
> limitations.
> 
> 
> 
> > 
> >