RE: [virtio-comment] Re: [PATCH v1 3/8] device-context: Define the device context fields for device migration

[Parav Pandit <parav@nvidia.com>]

> From: Jason Wang <jasowang@redhat.com>
> Sent: Friday, October 13, 2023 6:48 AM
> 
> On Thu, Oct 12, 2023 at 7:37âPM Parav Pandit <parav@nvidia.com> wrote:>
> > As Michael said, software based nesting is used..
> 
> I've pointed out in another thread when hardware has less abstraction level
> than nesting, trap/emulation is a must.
> 
> > See if actual hw based devices can implement it or not. Many components of
> cpu cannot do N level nesting either, but may be virtio can.
> > I donât know how yet.
> 
> I would not repeat the lessons given by Gerald J. Popek and Robert P.
> Goldberg[1] in 1976, but I think you miss a lot of fundamental things in the
> methodology of virtualization. 
Weekend is coming. I will read it.

> For example, nesting is a very important criteria
> to examine whether an architecture is well designed for virtualization.
>

In my reading of a leading OS vendor documentation, I leant that OS vendor do not recommend nested virtualization for production at [1].
Snippet:
"In addition, Red Hat does not recommend using nested virtualization in production user environments, due to various limitations in functionality. Instead, nested virtualization is primarily intended for development and testing scenarios."

[1] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_virtualization/creating-nested-virtual-machines_configuring-and-managing-virtualization

2nd leading hypervisor listed nested virtualization to be not used for "performance sensitive applications".

I want to repeat and emphasize that I am not ignoring the nested case.

An extension for nesting would be the VF presented to the guest itself with SR-IOV capability can work as_is as proposed here.
Michael presented the idea of the dummy PF, which is to represent the VF as dummy PF which can do the SR-IOV with one VF.
You need the support from the platform too, I guess TC can extend it.
May be a different interface more suitable for nested case which do not have performance needs.

How about a nested user to have AQ located on the VF so that mediation sw can operate admin commands over self?
Device mode commands will not be applicable there, instead some other things to be done.
So non passthrough mode software possibly can make use of it?

> That is to say for any CPU/hypervisor vendors, the architecture should be
> designed to run any levels of nesting instead of just an awkward 2 levels (but
> what you proposed can not work for even 2).
Huh, some missing text for corner case as making claim, _not_working in not a healthy discussion.

> For x86 and KVM, any level of
> nesting has been done for about 10 years ago.
>
I didnât find hw for PML support in x86 for N or 3 level nesting. Did I miss?
I didnât find hw for nested page tables upto N level walking on the PCIe read/writes in any cpu. Did I miss?
Have you seen nesting in hw works at N level?

> For virtio, it can do any level. So did for vhost/vDPA. For example, I usually
> develop and test virtio/vDPA/vhost in a nesting environment.
> 
Great.
Can you share the performance test results relative number with 2 and 3 level nesting covering the cpu utilization, latency?

> Thanks
> 
> [1] https://dl.acm.org/doi/pdf/10.1145/361011.361073