Re: [PATCH v1 3/8] device-context: Define the device context fields for device migration

["Michael S. Tsirkin" <mst@redhat.com>]

On Sun, Oct 08, 2023 at 02:25:50PM +0300, Parav Pandit wrote:
> Define the device context and its fields for purpose of device
> migration. The device context is read and written by the owner driver
> on source and destination hypervisor respectively.
> 
> Device context fields will experience a rapid growth post this initial
> version to cover many details of the device.
> 
> Fixes: https://github.com/oasis-tcs/virtio-spec/issues/176
> Signed-off-by: Parav Pandit <parav@nvidia.com>
> Signed-off-by: Satananda Burla <sburla@marvell.com>
> ---
> changelog:
> v0->v1:
> - enrich device context to cover feature bits, device configuration
>   fields
> - corrected alignment of device context fields
> ---
>  content.tex        |   1 +
>  device-context.tex | 142 +++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 143 insertions(+)
>  create mode 100644 device-context.tex
> 
> diff --git a/content.tex b/content.tex
> index 0a62dce..2698931 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -503,6 +503,7 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo
>  UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
>  
>  \input{admin.tex}
> +\input{device-context.tex}
>  
>  \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
>  
> diff --git a/device-context.tex b/device-context.tex
> new file mode 100644
> index 0000000..5611382
> --- /dev/null
> +++ b/device-context.tex
> @@ -0,0 +1,142 @@
> +\section{Device Context}\label{sec:Basic Facilities of a Virtio Device / Device Context}
> +
> +The device context holds the information that a owner driver can use
> +to setup a member device and resume its operation. The device context
> +of a member device is read or written by the owner driver using
> +administration commands.
> +
> +\begin{lstlisting}
> +struct virtio_dev_ctx_field_tlv {
> +        le32 type;
> +        le32 reserved;
> +        le64 length;
> +        u8 value[];
> +};
> +
> +struct virtio_dev_ctx {
> +        le32 field_count;
> +        struct virtio_dev_ctx_field_tlv fields[];
> +};
> +
> +\end{lstlisting}
> +
> +The \field{struct virtio_dev_ctx} is the device context of a member device.
> +The \field{field_count} indicates how many instances of
> +\field{struct virtio_dev_ctx_field_tlv} are present.
> +
> +The \field{struct virtio_dev_ctx_field_tlv} consist of \field{type} indicating
> +what data is contained in the \field{value} of length \field{length}.
> +The valid values for \field{type} can be found in the following table:
> +
> +\begin{tabularx}{\textwidth}{ |l||l|X| }
> +\hline
> +type & Name & Description \\
> +\hline \hline
> +0x0 & VIRTIO_DEV_CTX_PCI_COMMON_RUNTIME_CFG & Provides common configuration space of device for PCI transport \\
> +\hline
> +0x1 & VIRTIO_DEV_CTX_DEV_CFG_LAYOUT & Provides device specific configuration layout \\
> +\hline
> +0x2 & VIRTIO_DEV_CTX_DEV_FEATURES & Provides device features \\
> +\hline
> +0x3 & VIRTIO_DEV_CTX_PCI_VQ_CFG & Provides Virtqueue configuration for PCI transport \\
> +\hline
> +0x4 & VIRTIO_DEV_CTX_VQ_SPLIT_RUNTIME_CFG & Provides Queue run time state \\
> +\hline
> +0x5 & VIRTIO_DEV_CTX_VQ_SPLIT_DEV_OWN_DESC & Provides list of virtqueue descriptors owned by device  \\
> +\hline
> +0x6 - 0xFFFFFFFF & - & Reserved for future types \\
> +\hline
> +\end{tabularx}


I don't think this is enough, e.g. virtio net has internal state
controlled thought CVQ commands. how do you intend to address/migrate
these?

> +\subsubsection{Device Context Fields}\label{sec:Basic Facilities of a Virtio Device / Device Context / Device Context Fields}
> +
> +\paragraph{PCI Common Configuration Context}
> +\label{par:Basic Facilities of a Virtio Device / Device Context / Device Context Fields/ PCI Common Configuration Context}
> +
> +For the field VIRTIO_DEV_CTX_PCI_COMMON_RUNTIME_CFG, \field{type} is set to 0x0.
> +The \field{value} is in format of \field{struct virtio_pci_common_cfg}.
> +The \field{length} is the length of \field{struct virtio_pci_common_cfg}.
> +
> +\paragraph{Device Configuration Layout Context}
> +\label{par:Basic Facilities of a Virtio Device / Device Context / Device Context Fields/ Device Configuration Layout Context}
> +
> +For the field VIRTIO_DEV_CTX_DEV_CFG_LAYOUT, \field{type} is set to 0x1.
> +The \field{value} is in format of device specific configuration layout listed
> +in each of the device's device configuration layout section.
> +The \field{length} is the length of the device configuration layout data.

Unclear. I am guessing it's doing things like setting up RO
fields? This needs to be specified per device really.
Also how some fields behave might depend on features.

> +
> +\paragraph{Device Features Context}
> +\label{par:Basic Facilities of a Virtio Device / Device Context / Device Context Fields/ Device Features Context}
> +
> +For the field VIRTIO_DEV_CTX_DEV_FEATURES, \field{type} is set to 0x2.
> +The \field{value} is in format of device feature bits listed in
> +\ref{sec:Basic Facilities of a Virtio Device / Feature Bits} in the format of \field{struct virtio_dev_ctx_features}.
> +The \field{length} is the length of the device features.
> +
> +\begin{lstlisting}
> +struct virtio_dev_ctx_pci_vq_cfg {
> +        le64 feature_bits[];
> +};
> +\end{lstlisting}
> +
> +\paragraph{PCI Virtqueue Configuration Context}
> +\label{par:Basic Facilities of a Virtio Device / Device Context / Device Context Fields/ PCI Virtqueue Configuration Context}
> +
> +For the field VIRTIO_DEV_CTX_PCI_VQ_CFG, \field{type} is set to 0x3.
> +The \field{value} is in format of \field{struct virtio_dev_ctx_pci_vq_cfg}.
> +The \field{length} is the length of \field{struct virtio_dev_ctx_pci_vq_cfg}.
> +
> +\begin{lstlisting}
> +struct virtio_dev_ctx_pci_vq_cfg {
> +        le16 vq_index;
> +        le16 queue_size;
> +        le16 queue_msix_vector;
> +        le64 queue_descÍ
> +        le64 queue_driverÍ
> +        le64 queue_deviceÍ
> +};
> +\end{lstlisting}
> +
> +One or multiple entries of PCI Virtqueue Configuration Context may exist, each such
> +entry corresponds to a unique virtqueue identified by the \field{vq_index}.
> +
> +\paragraph{Virtqueue Split Mode Runtime Context}
> +\label{par:Basic Facilities of a Virtio Device / Device Context / Device Context Fields/ Virtqueue Split Mode Runtime Context}
> +
> +For the field VIRTIO_DEV_CTX_VQ_SPLIT_RUNTIME_CFG, \field{type} is set to 0x4.
> +The \field{value} is in format of \field{struct virtio_dev_ctx_vq_split_runtime}.
> +The \field{length} is the length of \field{struct virtio_dev_ctx_vq_split_runtime}.
> +
> +\begin{lstlisting}
> +struct virtio_dev_ctx_vq_split_runtime {
> +        le16 vq_index;
> +        le16 dev_avail_idx;
> +        u8 enabled;
> +};
> +\end{lstlisting}
> +
> +The \field{dev_avail_idx} indicates the next available index of the virtqueue from which
> +the device must start processing the available ring.
> +
> +One or multiple entries of Virtqueue Split Mode Runtime Context may exist, each such
> +entry corresponds to a unique virtqueue identified by the \field{vq_index}.
> +
> +\paragraph{Virtqueue Split Mode Device owned Descriptors Context}
> +
> +For the field VIRTIO_DEV_CTX_VQ_SPLIT_DEV_OWN_DESC, \field{type} is set to 0x5.
> +The \field{value} is in format of \field{struct virtio_dev_ctx_vq_split_runtime}.
> +The \field{length} is the length of \field{struct virtio_dev_ctx_vq_split_dev_descs}.
> +
> +\begin{lstlisting}
> +struct virtio_dev_ctx_vq_split_dev_descs {
> +        le16 vq_index;
> +        le16 desc_count;
> +        le16 desc_idx[];
> +};
> +\end{lstlisting}
> +
> +The \field{desc_idx} contains indices of the descriptors in \field{desc_count} of a
> +virtqueue identified by \field{vq_index} which is owned by the device.
> +
> +One or multiple entries of Virtqueue Split Mode Device owned Descriptors Context may exist, each such
> +entry corresponds to a unique virtqueue identified by the \field{vq_index}.
> -- 
> 2.34.1