RE: [virtio-comment] Re: [PATCH v1 3/8] device-context: Define the device context fields for device migration

[Parav Pandit <parav@nvidia.com>]

> From: Zhu, Lingshan <lingshan.zhu@intel.com>
> Sent: Monday, October 9, 2023 4:04 PM
> 
> On 10/8/2023 7:41 PM, Michael S. Tsirkin wrote:
> > On Sun, Oct 08, 2023 at 02:25:50PM +0300, Parav Pandit wrote:
> >> Define the device context and its fields for purpose of device
> >> migration. The device context is read and written by the owner driver
> >> on source and destination hypervisor respectively.
> >>
> >> Device context fields will experience a rapid growth post this
> >> initial version to cover many details of the device.
> >>
> >> Fixes: https://github.com/oasis-tcs/virtio-spec/issues/176
> >> Signed-off-by: Parav Pandit <parav@nvidia.com>
> >> Signed-off-by: Satananda Burla <sburla@marvell.com>
> >> ---
> >> changelog:
> >> v0->v1:
> >> - enrich device context to cover feature bits, device configuration
> >>    fields
> >> - corrected alignment of device context fields
> >> ---
> >>   content.tex        |   1 +
> >>   device-context.tex | 142
> +++++++++++++++++++++++++++++++++++++++++++++
> >>   2 files changed, 143 insertions(+)
> >>   create mode 100644 device-context.tex
> >>
> >> diff --git a/content.tex b/content.tex index 0a62dce..2698931 100644
> >> --- a/content.tex
> >> +++ b/content.tex
> >> @@ -503,6 +503,7 @@ \section{Exporting Objects}\label{sec:Basic Facilities
> of a Virtio Device / Expo
> >>   UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
> >>
> >>   \input{admin.tex}
> >> +\input{device-context.tex}
> >>
> >>   \chapter{General Initialization And Device
> >> Operation}\label{sec:General Initialization And Device Operation}
> >>
> >> diff --git a/device-context.tex b/device-context.tex new file mode
> >> 100644 index 0000000..5611382
> >> --- /dev/null
> >> +++ b/device-context.tex
> >> @@ -0,0 +1,142 @@
> >> +\section{Device Context}\label{sec:Basic Facilities of a Virtio
> >> +Device / Device Context}
> >> +
> >> +The device context holds the information that a owner driver can use
> >> +to setup a member device and resume its operation. The device
> >> +context of a member device is read or written by the owner driver
> >> +using administration commands.
> >> +
> >> +\begin{lstlisting}
> >> +struct virtio_dev_ctx_field_tlv {
> >> +        le32 type;
> >> +        le32 reserved;
> >> +        le64 length;
> >> +        u8 value[];
> >> +};
> >> +
> >> +struct virtio_dev_ctx {
> >> +        le32 field_count;
> >> +        struct virtio_dev_ctx_field_tlv fields[]; };
> >> +
> >> +\end{lstlisting}
> so this still doesn't work for nested

In one use case of nesting, that we came across is:
there is large host_VM which is hosting another guest_VMs.
In such case, the owner PF is passthrough to this host_VM and current proposed scheme continue to function for nesting as well for nested guest_VMs.

In second use case, where one want to bind only one member device to one VM, 
I think same plumbing can be extended to have another VF, to take the role of migration device instead of owner device.

I donât see a good way to passthrough and also do in-band migration without lot of device specific trap and emulation.
I also donât know the cpu performance numbers with 3 levels of nested page table translation which to my understanding cannot be accelerated by the current cpu.
Do you know how does it work for Intel x86_64?
Can it do > 2 level of nested page tables? If no, what is the perf characteristics to expect?