RE: [virtio-comment] Re: [PATCH v1 3/8] device-context: Define the device context fields for device migration

[Parav Pandit <parav@nvidia.com>]

> From: Zhu, Lingshan <lingshan.zhu@intel.com>
> Sent: Tuesday, October 24, 2023 4:00 PM
> 
> On 10/23/2023 6:14 PM, Parav Pandit wrote:
> >> From: Zhu, Lingshan <lingshan.zhu@intel.com>
> >> Sent: Monday, October 23, 2023 3:39 PM
> >>
> >> On 10/20/2023 8:54 PM, Parav Pandit wrote:
> >>>> From: Zhu, Lingshan <lingshan.zhu@intel.com>
> >>>> Sent: Friday, October 20, 2023 3:01 PM
> >>>>
> >>>> On 10/19/2023 6:33 PM, Parav Pandit wrote:
> >>>>>> From: Zhu, Lingshan <lingshan.zhu@intel.com>
> >>>>>> Sent: Thursday, October 19, 2023 2:48 PM
> >>>>>>
> >>>>>> On 10/19/2023 5:14 PM, Michael S. Tsirkin wrote:
> >>>>>>> On Thu, Oct 19, 2023 at 09:13:16AM +0000, Parav Pandit wrote:
> >>>>>>>>> Oh, really? Quite interesting, do you want to move all config
> >>>>>>>>> space fields in VF to admin vq? Have a plan?
> >>>>>>>> Not in my plan for spec 1.4 time frame.
> >>>>>>>> I do not want to divert the discussion, would like to focus on
> >>>>>>>> device
> >>>>>> migration phases.
> >>>>>>>> Lets please discuss in some other dedicated thread.
> >>>>>>> Possibly, if there's a way to send admin commands to vf itself
> >>>>>>> then Lingshan will be happy?
> >>>>>> still need to prove why admin commands are better than registers.
> >>>>> Virtio spec development is not proof based approach. Please stop
> >>>>> asking for
> >> it.
> >>>>> I tried my best to have technical answer in [1].
> >>>>> I explained that registers simply do not work for passthrough mode
> >>>>> (if this is what you are asking when you are asking prove its better).
> >>>>> They can work for non_passthrough mediated mode.
> >>>>>
> >>>>> A member device may do admin commands using registers. Michael and
> >>>>> I are
> >>>> discussing presently in the same thread.
> >>>>> Since there are multiple things to be done for device migration,
> >>>>> dedicated
> >>>> register set for each functionality do not scale well, hard to
> >>>> maintain and extend.
> >>>>> A register holding a command content make sense.
> >>>>>
> >>>>> Now, with that, if this can be useful only for non_passthrough, I
> >>>>> made humble
> >>>> request to transport them using AQ, this way, you get all benefits of AQ.
> >>>>> And trying to understand, why AQ cannot possible or inferior?
> >>>>>
> >>>>> If you have commands like suspend/resume device, register or queue
> >>>> transport simply donât work, because it's wrong to bifurcate the
> >>>> device with such weird API.
> >>>>> If you want to biferacate for mediation software, it probably
> >>>>> makes sense to
> >>>> operate at each VQ level, config space level. Such are very
> >>>> different commands than passthrough.
> >>>>> I think vdpa has demonstrated that very well on how to do specific
> >>>>> work for
> >>>> specific device type. So some of those work can be done using AQ.
> >>>>> [1]
> >>>>> https://lore.kernel.org/virtio-comment/870ace02-f99c-4582-932f-bd1
> >>>>> 03
> >>>>> 36
> >>>>> 2dae9@intel.com/T/#m37743aa924536d0256d6b3b8e83a11c750f28794
> >>>> We have been through your statement for many times.
> >>>> This is not about how many times you repeated, if you think this is
> >>>> true, you need to prove that with solid evidence.
> >>>>
> >>> I will not respond to this comment anymore.
> >> Ok if you choose not to respond.
> >>>> For pass-through, I still recommend you to take a reference of
> >>>> current virito-pci implementation, it works for pass-through, right?
> >>> What do you mean by current virtio-pci implementation?
> >> current virito-pci works for pass-through
> > I still donât understand what is "current virtio-pci".
> > Do you mean qemu implementation of emulated virtio-pci or you mean
> virtio-pci specification for passthrough?
> > What do you want me to refer to for passthrough? Please clarify.
> you know guest vcpu and its vRC can not access host side devices, and there
> must be a driver helping the pass-through use cases, like vDPA and vfio
I am not sure how to corelate this answer to the question of "virtio-pci for passthrough".
:(

Today when a virtio-pci member device is passthrough to the guest VM, hypervisor is not involved in virtio interface such as config space, cvq, data vq etc.
Do you agree?

> >>>> For scale, I already told you for many times that they are
> >>>> per-device facilities. How can a per-device facility not scale?
> >>> Each VF device must implement new set of on-chip memory-based
> >>> registers
> >> which demands more power, die area which does not scale efficiently
> >> to thousands of VFs.
> >> that can be fpga gates or SOC implementing new features, you think
> >> that is a waste?
> > It is waste in hw, if there is a better approach possible to not burn them as
> gates and save on resources for rarely used items.
> Is a new entry in MSIX table a waste of HW?
Not as must as existing MSI-X table entries which requires linear amount of on-chip memory.

> Can I say implementing admin vq in SOC is a waste of cores?
Which cores in the SoC?
If it is on the PF, there is only handful of AQs for scale of N VFs.

> >
> >
> >>>> vDPA works fine on config space.
> >>>>
> >>>> So, if you still insist admin vq is better than config space like
> >>>> in other thread you have concluded, you may imply that config space
> >>>> interfaces should be re-factored to admin vq.
> >>> Whatever is done in past is done, there is no way to change history.
> >>> An new non init time registers should not be placed in device
> >>> specific config
> >> space as virtio spec has clear guideline on it for good.
> >>> Device context reading, dirty page address reading, changing vf
> >>> device modes,
> >> all of these are clearly not a init time settings.
> >>> Hence, they do not belong to the registers.
> >> reset vq? and you get it from Appendix B. Creating New Device Types,
> >> are we implementing a new type of device???
> > I donât understand your question.
> > I replied the history of reset_vq.
> > Take good examples to follow, reset_vq clearly is not the one.
> so again, we are not implementing new device type, so your citation doesn't
> apply.
I disagree.
I am engineer to build practical systems considering limitations and also advancements of the transport; while listening to other industry efforts,
I am no from legal department.
Hence, Appendix B makes a sense to me to apply to the existing device which also has the section for "device improvements".