Re: [virtio-comment] Re: [PATCH v1 3/8] device-context: Define the device context fields for device migration

["Zhu, Lingshan" <lingshan.zhu@intel.com>]

On 10/9/2023 10:30 PM, Parav Pandit wrote:
> > From: Zhu, Lingshan <lingshan.zhu@intel.com>
> > Sent: Monday, October 9, 2023 4:04 PM
> >
> > On 10/8/2023 7:41 PM, Michael S. Tsirkin wrote:
> > > On Sun, Oct 08, 2023 at 02:25:50PM +0300, Parav Pandit wrote:
> > > > Define the device context and its fields for purpose of device
> > > > migration. The device context is read and written by the owner driver
> > > > on source and destination hypervisor respectively.
> > > >
> > > > Device context fields will experience a rapid growth post this
> > > > initial version to cover many details of the device.
> > > >
> > > > Fixes: https://github.com/oasis-tcs/virtio-spec/issues/176
> > > > Signed-off-by: Parav Pandit <parav@nvidia.com>
> > > > Signed-off-by: Satananda Burla <sburla@marvell.com>
> > > > ---
> > > > changelog:
> > > > v0->v1:
> > > > - enrich device context to cover feature bits, device configuration
> > > >     fields
> > > > - corrected alignment of device context fields
> > > > ---
> > > >    content.tex        |   1 +
> > > >    device-context.tex | 142
> > +++++++++++++++++++++++++++++++++++++++++++++
> > > >    2 files changed, 143 insertions(+)
> > > >    create mode 100644 device-context.tex
> > > >
> > > > diff --git a/content.tex b/content.tex index 0a62dce..2698931 100644
> > > > --- a/content.tex
> > > > +++ b/content.tex
> > > > @@ -503,6 +503,7 @@ \section{Exporting Objects}\label{sec:Basic Facilities
> > of a Virtio Device / Expo
> > > >    UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
> > > >
> > > >    \input{admin.tex}
> > > > +\input{device-context.tex}
> > > >
> > > >    \chapter{General Initialization And Device
> > > > Operation}\label{sec:General Initialization And Device Operation}
> > > >
> > > > diff --git a/device-context.tex b/device-context.tex new file mode
> > > > 100644 index 0000000..5611382
> > > > --- /dev/null
> > > > +++ b/device-context.tex
> > > > @@ -0,0 +1,142 @@
> > > > +\section{Device Context}\label{sec:Basic Facilities of a Virtio
> > > > +Device / Device Context}
> > > > +
> > > > +The device context holds the information that a owner driver can use
> > > > +to setup a member device and resume its operation. The device
> > > > +context of a member device is read or written by the owner driver
> > > > +using administration commands.
> > > > +
> > > > +\begin{lstlisting}
> > > > +struct virtio_dev_ctx_field_tlv {
> > > > +        le32 type;
> > > > +        le32 reserved;
> > > > +        le64 length;
> > > > +        u8 value[];
> > > > +};
> > > > +
> > > > +struct virtio_dev_ctx {
> > > > +        le32 field_count;
> > > > +        struct virtio_dev_ctx_field_tlv fields[]; };
> > > > +
> > > > +\end{lstlisting}
> > so this still doesn't work for nested
> In one use case of nesting, that we came across is:
> there is large host_VM which is hosting another guest_VMs.
> In such case, the owner PF is passthrough to this host_VM and current proposed scheme continue to function for nesting as well for nested guest_VMs.
The system admin can choose only passthrough some of the devices for 
nested guests, so passthrough the PF to L1 guest is not a good idea, 
because there can be
many devices still work for the host or L1.
> In second use case, where one want to bind only one member device to one VM,
> I think same plumbing can be extended to have another VF, to take the role of migration device instead of owner device.
>
> I donât see a good way to passthrough and also do in-band migration without lot of device specific trap and emulation.
> I also donât know the cpu performance numbers with 3 levels of nested page table translation which to my understanding cannot be accelerated by the current cpu.
host_PA->L1_QEMU_VA->L1_Guest_PA->L1_QEMU_VA->L2_Guest_PA and so on, 
there can be performance overhead, but can be done.

So admin vq migration still don't work for nested, this is surely a blocker.
> Do you know how does it work for Intel x86_64?
> Can it do > 2 level of nested page tables? If no, what is the perf characteristics to expect?
of course that can be done, Page table is not a problem, there are soft 
mmu emulation and viommu, through performance overhead.