OASIS Mailing List ArchivesView the OASIS mailing list archive below
or browse/search using MarkMail.

 


Help: OASIS Mailing Lists Help | MarkMail Help

virtio-comment message

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]

Subject: Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification


On 8/2/2021 5:25 AM, Jason Wang wrote:


å 2021/7/28 äå8:48, Michael S. Tsirkin åé:

On Mon, Jul 26, 2021 at 07:52:53PM +0300, Max Gurtovoy wrote:

Admin virtqueues will be used to send administrative commands to
manipulate various features of the device which would not easily map
into the configuration space.

The same Admin command format will be used for all virtio devices. The
Admin command set will include 4 types of command classes:
1. The generic common class
2. The transport specific class
3. The device specific class
4. The vendor specific class

The above mechanism will enable adding various features to the virtio
specification, e.g.:
1. Format virtio-blk devices in various configurations (512B block size, 
ÂÂÂ 512B + 8B T10-DIF, 4K block size, 4k + 8B T10-DIF, etc..).
2. Live migration management.
3. Encrypt/Decrypt descriptors.
4. Virtualization management.
5. Get device error logs.
6. Implement advanced vendor/device/transport specific features.
7. Run device health test.
8. More.
As virtio evolves beyond the para-virt/sw-emulated world, it's mandatory 
for the specification to become flexible and allow a wider feature set.
The corrent ctrl virtq that is defined for some of the virtio devices is 
device specific and wasn't designed to be a generic virtq for
admininistration.

Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>

Lots of things on this list seem to make sense when
done from PF and affecting a VF.
I think from this POV the generic structure should include
a way to address one device from another.



---
 admin-virtq.tex | 241 ++++++++++++++++++++++++++++++++++++++++++++++++ 
 content.tex | 4 +
 2 files changed, 245 insertions(+)
 create mode 100644 admin-virtq.tex

diff --git a/admin-virtq.tex b/admin-virtq.tex
new file mode 100644
index 0000000..ccec2ca
--- /dev/null
+++ b/admin-virtq.tex
@@ -0,0 +1,241 @@
+\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues} 
+
+Admin virtqueues are used to send administrative commands to manipulate 
+various features of the device which would not easily map into the
+configuration space.
+
+Use of Admin virtqueues is negotiated by the VIRTIO_F_ADMIN_VQ
+feature bit.
+
+Admin virtqueue index may vary among different device types.
+
+All commands are of the following form:
+
+\begin{lstlisting}
+struct virtio_admin_cmd {
+ÂÂÂÂÂÂÂ /* Device-readable part */
+ÂÂÂÂÂÂÂ u8 class;
+ÂÂÂÂÂÂÂ u8 command;
+ÂÂÂÂÂÂÂ u8 command-specific-data[];
+
+ÂÂÂÂÂÂÂ /* Device-writable part */
+ÂÂÂÂÂÂÂ u8 command-specific-result[];
+ÂÂÂÂÂÂÂ u8 status_type : 4;
+ÂÂÂÂÂÂÂ u8 reserved : 4;
+ÂÂÂÂÂÂÂ u8 status;
+};
+
+/* Status type values */
+#define VIRTIO_ADMIN_STATUS_TYPE_GENERICÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 0
+#define VIRTIO_ADMIN_STATUS_TYPE_CLASS_SPECIFICÂÂÂÂÂÂÂ 1
+#define VIRTIO_ADMIN_STATUS_TYPE_COMMAND_SPECIFICÂÂÂÂÂ 2
+#define VIRTIO_ADMIN_STATUS_TYPE_TRANSPORT_SPECIFICÂÂÂ 3
+#define VIRTIO_ADMIN_STATUS_TYPE_DEVICE_SPECIFICÂÂÂÂÂÂ 4
+#define VIRTIO_ADMIN_STATUS_TYPE_VENDOR_SPECIFICÂÂÂÂÂÂ 5
+
+/* Generic status values */
+#define VIRTIO_ADMIN_STATUS_GENERIC_OKÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 0
+#define VIRTIO_ADMIN_STATUS_GENERIC_ERRÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 1
+#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_CLASSÂÂÂÂÂÂÂÂÂ 2
+#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_COMMANDÂÂÂÂÂÂÂ 3
+#define VIRTIO_ADMIN_STATUS_GENERIC_DATA_TRANSFER_ERRÂÂÂÂÂ 4
+#define VIRTIO_ADMIN_STATUS_GENERIC_DEVICE_INTERNAL_ERRÂÂÂ 5
+\end{lstlisting}
+
+The \field{class}, \field{command} and \field{command-specific-data} are 
+set by the driver, and the device sets the \field{status_type}, the
+\field{status} and the \field{command-specific-result}, if needed.
+
+The virtio Admin command class codes are divided in the following form: 
+
+\begin{lstlisting}
+/* class values that are transport, device and vendor independent */
+#define VIRTIO_ADMIN_COMMON_CLASS_STARTÂÂÂ 0
+#define VIRTIO_ADMIN_COMMON_CLASS_ENDÂÂÂÂÂ 63
+
+/* class values that are transport specific */
+#define VIRTIO_ADMIN_TRANSPORT_CLASS_STARTÂ 64
+#define VIRTIO_ADMIN_TRANSPORT_CLASS_ENDÂÂÂ 127
+
+/* class values that are device specific */
+#define VIRTIO_ADMIN_DEVICE_CLASS_STARTÂÂÂÂ 128
+#define VIRTIO_ADMIN_DEVICE_CLASS_ENDÂÂÂÂÂÂ 191
+
+/* class values that are vendor specific */
+#define VIRTIO_ADMIN_VENDOR_CLASS_STARTÂÂÂÂ 192
+#define VIRTIO_ADMIN_VENDOR_CLASS_ENDÂÂÂÂÂÂ 255
+\end{lstlisting}
+
+\subsection{Admin command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set} 
+
+Each virtio device that advertise VIRTIO_F_ADMIN_VQ feature, MUST
+support all the mandatory admin commands. A device MAY support also
+one or more optional admin commands.
+
+\subsubsection{Common command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set} 
+
+The Common command set is a group of classes and commands within each
+of these classes which are transport, device and vendor independent.
+A mandatory class is a class that has at least one mandatory command.
+The Common command set is summarized in following table:
+
+\begin{tabular}{|l|l|l|}
+\hline
+Class & Description & M/O \\
+\hline \hline
+0Â & VIRTIO_ADMIN_DISCOVER_DEVICEÂÂÂ & M \\
+\hline
+1Â & VIRTIO_ADMIN_DISCOVER_DEVICE_CLASS_COMMANDSÂÂÂ & M \\
+\hline
+2-63Â & reservedÂÂÂ & - \\
+\hline
+\end{tabular}
+
+\paragraph{Discover device class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class} 
+
+This class (opcode: 0) of commands is used to query generic device
+information. The following table describes the commands supported for
+this class:
+
+\begin{tabular}{|l|l|l|}
+\hline
+Command & Description & M/O \\
+\hline \hline
+0Â & VIRTIO_ADMIN_DISCOVER_DEVICE_IDENTITYÂÂÂ & M \\
+\hline
+1Â & VIRTIO_ADMIN_DISCOVER_DEVICE_SUPPORTED_CLASSES & M \\
+\hline
+2-255Â & reservedÂÂÂ & - \\
+\hline
+\end{tabular}

So there are several problems with this approach.
One is that there is no two way negotiation.
No way for device to know what will driver use in the end.
This breaks things like e.g. accelerating some configurations
but not others.

Another is that everything is going through the admin vq.
Hard for hypervisor to intervene and change just some aspects
of the device.

This is also a problem for validation, tricky
to find out during probe what does device support and whether
driver can work with it.
Yes, so I think it makes sense to make the admin virtqueue as a transport and then build other stuffs on top. 


what transport exactly ?

it should be a virtq for devices to open.








Can't we map this stuff into memory instead?
Maybe have an admin config structure
that is separate from regular config?
Only issue we can't then make it too big.
But so far I don't see a lot of info used, most is reserved.



It should work (especially for PCI).


you're trying to invent some mechanism that is not flexible and non trivial.

You have already a virtq - so lets use it.





Thanks







+
+\subparagraph{Device identity command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device identity command} 
+
+This mandatory command should return device identity in the following
+structure:
+
+\begin{tabular}{|l|l|l|}
+\hline
+Bytes & Description & M/O \\
+\hline \hline
+03:00Â & VIRTIO DEVICE IDÂÂÂ & M \\
+\hline
+05:04Â & VIRTIO TRANSPORT IDÂÂÂ & M \\
+\hline
+4095:06Â & reservedÂÂÂ & - \\
+ \hline
+\end{tabular}
+
+\subparagraph{Device supported classes command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device supported classes command} 
+
+This mandatory command should return device identity in the following
+structure:
+
+\begin{tabular}{|l|l|l|}
+\hline
+Bytes & Description & M/O \\
+\hline \hline
+03:00Â & VIRTIO_ADMIN_CLASS_0_PROPERTIESÂÂÂ & M \\
+\hline
+07:04Â & VIRTIO_ADMIN_CLASS_1_PROPERTIESÂÂÂ & M \\
+ \hline
+11:08Â & VIRTIO_ADMIN_CLASS_2_PROPERTIESÂÂÂ & M \\
+\hline
+... & ... & M \\
+\hline
+1023:1020Â & VIRTIO_ADMIN_CLASS_255_PROPERTIESÂÂÂ & M \\
+\hline
+4095:1024Â & reservedÂÂÂ & - \\
+\hline
+\end{tabular}
+
+The above structure is divided to 256 sections of 4B each, that will
+describe whether a device supports a certain class code. The
+following 3072B are reserved. each of the non-reserved 4B sections will 
+follow the following data structure:
+
+\begin{tabular}{|l|l|l|}
+\hline
+Bits & Description & M/O \\
+\hline \hline
+0Â & SUPPORTED (1: yes, 0:no)ÂÂÂ & M \\
+\hline
+31:01Â & reservedÂÂÂ & - \\
+ \hline
+\end{tabular}
+
+\paragraph{Discover device class commands class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class commands class} 
+
+This class (opcode: 1) of commands is used to query supported commands
+for each supported device class.
The following table describes the commands
+supported for this class:
+
+\begin{tabular}{|l|l|l|}
+\hline
+Command & Description & M/O \\
+\hline \hline
+0Â & VIRTIO_ADMIN_CLASS_0_COMMANDS_IDENTITYÂÂÂ & M \\
+\hline
+1Â & VIRTIO_ADMIN_CLASS_1_COMMANDS_IDENTITYÂÂÂ & M \\
+\hline
+2Â & VIRTIO_ADMIN_CLASS_2_COMMANDS_IDENTITYÂÂÂ & M \\
+\hline
+... & ... & M \\
+\hline
+255Â & VIRTIO_ADMIN_CLASS_255_COMMANDS_IDENTITYÂÂÂ & M \\
+\hline
+\end{tabular}
+
+Each command in this class, will return class identity in the following structure: 
+
+\begin{tabular}{|l|l|l|}
+\hline
+Bytes & Description & M/O \\
+\hline \hline
+03:00Â & VIRTIO_ADMIN_COMMAND_0_PROPERTIESÂÂÂ & M \\
+\hline
+07:04Â & VIRTIO_ADMIN_COMMAND_1_PROPERTIESÂÂÂ & M \\
+ \hline
+11:08Â & VIRTIO_ADMIN_COMMAND_2_PROPERTIESÂÂÂ & M \\
+\hline
+... & ... & M \\
+\hline
+1023:1020Â & VIRTIO_ADMIN_COMMAND_255_PROPERTIESÂÂÂ & M \\
+\hline
+4095:1024Â & reservedÂÂÂ & - \\
+\hline
+\end{tabular}
+
+The above structure is divided to 256 sections of 4B each, that will
+describe whether a class supports a certain command code. The
+following 3072B are reserved. each of the non-reserved 4B sections will 
+follow the following data structure:
+
+\begin{tabular}{|l|l|l|}
+\hline
+Bits & Description & M/O \\
+\hline \hline
+0Â & SUPPORTED (1: yes, 0:no)ÂÂÂ & M \\
+\hline
+31:01Â & reservedÂÂÂ & - \\
+ \hline
+\end{tabular}
+
+\subsubsection{Transport command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Transport command set} 
+
+The Transport command set is a group of classes and commands within
+each of these classes which are transport specific. Refer to each
+transport section for more information on the supported commands.
+
+\subsubsection{Device command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Device command set} 
+
+The Device command set is a group of classes and commands within
+each of these classes which are device specific. Refer to each
+device section for more information on the supported commands.
+
+\subsubsection{Vendor specific command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Vendor specific command set} 
+
+The Vendor specific command set is a group of classes and commands
+within each of these classes which are vendor specific. Refer to
+each vendor specification for more information on the supported
+commands.

Here's another example.
It's important that there is a way to make a device completely
generic without vendor specific expensions.
Would be hard to do here.

That's a generic comment.

but specifically I am very reluctant to add vendor specific stuff like
this directly in the spec. We already have VIRTIO_PCI_CAP_VENDOR_CFG
and if that is not sufficient I would like to know why
before we add more vendor specific stuff.



diff --git a/content.tex b/content.tex
index c266fd5..d350175 100644
--- a/content.tex
+++ b/content.tex
@@ -407,6 +407,8 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo 
 types. It is RECOMMENDED that devices generate version 4
 UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
 +\input{admin-virtq.tex}
+
 \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}   We start with an overview of device initialization, then expand on the @@ -6671,6 +6673,8 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}  data to be provided in driver notification and the delivery method is 
ÂÂÂ transport specific.
ÂÂÂ For more details about driver notifications over PCI see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Available Buffer Notifications}. 
+Â \item[VIRTIO_F_ADMIN_VQ(40)] This feature indicates that
+Â the device supports administration virtqueue negotiation.
  \end{description}
 --
2.21.0

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]