Re: [virtio-comment] Re: [PATCH v1 3/8] device-context: Define the device context fields for device migration

[Jason Wang <jasowang@redhat.com>]

On Fri, Oct 13, 2023 at 2:40âPM Parav Pandit <parav@nvidia.com> wrote:
>
>
> > From: Jason Wang <jasowang@redhat.com>
> > Sent: Friday, October 13, 2023 6:48 AM
> >
> > On Thu, Oct 12, 2023 at 7:37âPM Parav Pandit <parav@nvidia.com> wrote:>
> > > As Michael said, software based nesting is used..
> >
> > I've pointed out in another thread when hardware has less abstraction level
> > than nesting, trap/emulation is a must.
> >
> > > See if actual hw based devices can implement it or not. Many components of
> > cpu cannot do N level nesting either, but may be virtio can.
> > > I donât know how yet.
> >
> > I would not repeat the lessons given by Gerald J. Popek and Robert P.
> > Goldberg[1] in 1976, but I think you miss a lot of fundamental things in the
> > methodology of virtualization.
> Weekend is coming. I will read it.
>
> > For example, nesting is a very important criteria
> > to examine whether an architecture is well designed for virtualization.
> >
>
> In my reading of a leading OS vendor documentation, I leant that OS vendor do not recommend nested virtualization for production at [1].
> Snippet:
> "In addition, Red Hat does not recommend using nested virtualization in production user environments, due to various limitations in functionality. Instead, nested virtualization is primarily intended for development and testing scenarios."
>
> [1] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_virtualization/creating-nested-virtual-machines_configuring-and-managing-virtualization
>
> 2nd leading hypervisor listed nested virtualization to be not used for "performance sensitive applications".

Another concept shift.

I'm not going to comment on the choice for individual distros. But the
points are whether we can deploy a nesting virtualization easily under
a specific hardware architecture. In this regard, the above is a good
example.

Again, just a simple google will tell you the instances that support
nesting have been available for almost all the major cloud vendors for
a while.

>
> I want to repeat and emphasize that I am not ignoring the nested case.
>
> An extension for nesting would be the VF presented to the guest itself with SR-IOV capability can work as_is as proposed here.

How can a VF have the SR-IOV capability?

> Michael presented the idea of the dummy PF, which is to represent the VF as dummy PF which can do the SR-IOV with one VF.

Why do we need the complicated SR-IOV emulation at the nesting level?
How can you make sure such a design can result in a live migration to
be done at any levels?

E.g in LN, you had a PF and a VF. How to migrate the PF to this level?
You want two PFs in the L(N-1) level?

> You need the support from the platform too, I guess TC can extend it.
> May be a different interface more suitable for nested case which do not have performance needs.

I disagree, it's about if the performance can satisfy the requirement
at N level.

>
> How about a nested user to have AQ located on the VF so that mediation sw can operate admin commands over self?

I would go with such complicated architecture.

> Device mode commands will not be applicable there, instead some other things to be done.
> So non passthrough mode software possibly can make use of it?

It would be a great burden if you

1) use passthrough in L0
2) use trap/emulation in L(N+1)

>
> > That is to say for any CPU/hypervisor vendors, the architecture should be
> > designed to run any levels of nesting instead of just an awkward 2 levels (but
> > what you proposed can not work for even 2).
> Huh, some missing text for corner case as making claim, _not_working in not a healthy discussion.
>
> > For x86 and KVM, any level of
> > nesting has been done for about 10 years ago.
> >
> I didnât find hw for PML support in x86 for N or 3 level nesting. Did I miss?
> I didnât find hw for nested page tables upto N level walking on the PCIe read/writes in any cpu. Did I miss?

You need first asking why it is a must to achieve nested
virtualization. All of those obstacles come only if you want to use
"passthrough" for any levels.

> Have you seen nesting in hw works at N level?

Again, hardware can't have endless resources for endless levels. Trap
and emulation is a must for achieving nesting virtualization. If you
try to invent a passthrough method that can work for any level, you
will probably fail

Thanks

>
> > For virtio, it can do any level. So did for vhost/vDPA. For example, I usually
> > develop and test virtio/vDPA/vhost in a nesting environment.
> >
> Great.
> Can you share the performance test results relative number with 2 and 3 level nesting covering the cpu utilization, latency?
>
> > Thanks
> >
> > [1] https://dl.acm.org/doi/pdf/10.1145/361011.361073