[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]
Subject: Re: [PATCH v6 3/4] admin: Add group member legacy register access commands
On Tue, Jun 13, 2023 at 08:30:14PM +0300, Parav Pandit wrote:
> Introduce group member legacy common configuration and legacy device
> configuration access read/write commands.
>
> Group member legacy registers access commands enable group owner driver
> software to access legacy registers on behalf of the guest virtual
> machine.
>
> Usecase:
> ========
> 1. A hypervisor/system needs to provide transitional
> virtio devices to the guest VM at scale of thousands,
> typically, one to eight devices per VM.
>
> 2. A hypervisor/system needs to provide such devices using a
> vendor agnostic driver in the hypervisor system.
>
> 3. A hypervisor system prefers to have single stack regardless of
> virtio device type (net/blk) and be future compatible with a
> single vfio stack using SR-IOV or other scalable device
> virtualization technology to map PCI devices to the guest VM.
> (as transitional or otherwise)
>
> Motivation/Background:
> =====================
> The existing virtio transitional PCI device is missing support for
> PCI SR-IOV based devices. Currently it does not work beyond
> PCI PF, or as software emulated device in reality. Currently it
> has below cited system level limitations:
>
> [a] PCIe spec citation:
> VFs do not support I/O Space and thus VF BARs shall not indicate I/O Space.
>
> [b] cpu arch citiation:
> Intel 64 and IA-32 Architectures Software Developerâs Manual:
> The processorâs I/O address space is separate and distinct from
> the physical-memory address space. The I/O address space consists
> of 64K individually addressable 8-bit I/O ports, numbered 0 through FFFFH.
>
> [c] PCIe spec citation:
> If a bridge implements an I/O address range,...I/O address range will be
> aligned to a 4 KB boundary.
>
> Overview:
> =========
> Above usecase requirements is solved by PCI PF group owner accessing
> its group member PCI VFs legacy registers using an admin virtqueue of
> the group owner PCI PF.
>
> Two types of administration commands are added which read/write PCI VF
> registers.
>
> Software usage example:
> =======================
>
> 1. One way to use and map to the guest VM is by using vfio driver
> framework in Linux kernel.
>
> +----------------------+
> |pci_dev_id = 0x100X |
> +---------------|pci_rev_id = 0x0 |-----+
> |vfio device |BAR0 = I/O region | |
> | |Other attributes | |
> | +----------------------+ |
> | |
> + +--------------+ +-----------------+ |
> | |I/O BAR to AQ | | Other vfio | |
> | |rd/wr mapper& | | functionalities | |
> | | forwarder | | | |
> | +--------------+ +-----------------+ |
> | |
> +------+-------------------------+-----------+
> | |
> Config region |
> access Driver notifications
> | |
> +----+------------+ +----+------------+
> | +-----+ | | PCI VF device A |
> | | AQ |-------------+---->+-------------+ |
> | +-----+ | | | | legacy regs | |
> | PCI PF device | | | +-------------+ |
> +-----------------+ | +-----------------+
> |
> | +----+------------+
> | | PCI VF device N |
> +---->+-------------+ |
> | | legacy regs | |
> | +-------------+ |
> +-----------------+
>
> 2. Continue to use the virtio pci driver to bind to the
> listed device id and use it as in the host.
>
> 3. Use it in a light weight hypervisor to run bare-metal OS.
>
> Fixes: https://github.com/oasis-tcs/virtio-spec/issues/167
> Signed-off-by: Parav Pandit <parav@nvidia.com>
> ---
> changelog:
> v5->v6:
> - fixed previous missed abbreviation of LCC and LD
> v4->v5:
> - split from pci transport specific patch
> - split conformance to transport and generic sections
> - written the description of the command as generic with member
> and group device terminology
> - reflected many section names to remove VF
> - rename fields from register to region
> - avoided abbreviation for legacy, device and config
> ---
> admin-cmds-legacy-access.tex | 138 +++++++++++++++++++++++++++++++++++
> admin.tex | 12 ++-
> conformance.tex | 2 +
> 3 files changed, 151 insertions(+), 1 deletion(-)
> create mode 100644 admin-cmds-legacy-access.tex
>
> diff --git a/admin-cmds-legacy-access.tex b/admin-cmds-legacy-access.tex
> new file mode 100644
> index 0000000..89a4f49
> --- /dev/null
> +++ b/admin-cmds-legacy-access.tex
> @@ -0,0 +1,138 @@
> +\subsubsection{Legacy Interfaces: legacy configuration access commands}\label{sec:Basic Facilities of a Virtio Device / Device groups / Group
> +administration commands / Legacy Interfaces: legacy configuration access commands}
I think this is where all the theory of operation text should go.
> +
> +The group owner device can optionally enable the driver to access its member devices legacy common
> +configuration and device configuration region
we can just say "the legacy interface of member
devices"
> using following listed
using the following
> +administrative commands:
administration not administrative
> +
> +\begin{enumerate}
> +\item Legacy Common Configuration Region Write Command
> +\item Legacy common Configuration Region Read Command
> +\item Legacy Device Configuration Region Write Command
> +\item Legacy Device Configuration Region Read Command
> +\end{enumerate}
> +
> +\paragraph{Legacy Common Configuration Region Write Command}\label{par:Basic Facilities of a Virtio Device / Device groups / Group
> +administration commands / Legacy Interfaces: legacy configuration access commands / Legacy Common Configuration Region Write Command}
> +
> +This command writes in a legacy common configuration region of a group member device.
> +This command follows \field{struct virtio_admin_cmd}.
> +This command uses following structure for \field{command_specific_data}:
> +
> +\begin{lstlisting}
> +struct virtio_admin_cmd_legacy_common_cfg_wr_data {
> + u8 offset; /* Starting byte offset of the common configuration region to write */
> + u8 region_data[];
> +};
> +\end{lstlisting}
> +
> +The driver sets command \field{opcode} to VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE.
> +The driver sets valid \field{offset} and associated \field{region_data} bytes to
> +write to the common configuration region.
> +
> +This command does not have any command specific result.
> +
> +\paragraph{Legacy Common Configuration Region Read Command}\label{par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interfaces: legacy configuration access commands / Legacy Common Configuration Region Read Command}
> +
> +This command reads from a legacy common configuration region of a group member device.
> +This command follows \field{struct virtio_admin_cmd}.
> +This command uses following listed structure for \field{command_specific_data}:
> +
> +\begin{lstlisting}
> +struct virtio_admin_cmd_legacy_common_cfg_rd_data {
> + u8 offset; /* Starting byte offset of the common configuration region to read */
> +};
> +\end{lstlisting}
> +
> +The driver sets command \field{opcode} to VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ.
> +
> +The driver sets valid \field{offset} of the region from where to read
> +\field{region_data}.
> +
> +When command completes successfully, \field{command_specific_result}
> +uses following listed structure:
> +
> +\begin{lstlisting}
> +struct virtio_admin_cmd_legacy_common_cfg_rd_result {
> + u8 region_data[];
> +};
> +\end{lstlisting}
> +
> +\paragraph{Legacy Device Configuration Region Write Command}\label{par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interfaces: legacy configuration access commands / Legacy Device Configuration Region Write Command}
> +
> +This command writes in a legacy device configuration registers of a group member device.
> +This command follows \field{struct virtio_admin_cmd}.
> +This command uses following structure for \field{command_specific_data}:
> +
> +\begin{lstlisting}
> +struct virtio_admin_cmd_ld_reg_wr_data {
> + u8 offset; /* Starting byte offset of the device configuration region to write */
> + u8 region_data[];
> +};
> +\end{lstlisting}
> +
> +The driver sets command \field{opcode} to VIRTIO_ADMIN_CMD_LEGACY_DEV_REG_WRITE.
> +The driver sets valid \field{offset} and associated \field{region_data} bytes to
> +the device configuration region.
> +
> +This command does not have any command specific result.
> +
> +\paragraph{Legacy Device Registers Read Command}\label{par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interfaces: legacy configuration access commands / Legacy Device Configuration Region Read Command}
> +
> +This command reads from a legacy device configuration region of a group member device.
> +This command follows \field{struct virtio_admin_cmd}.
> +
> +This command uses following listed structure for \field{command_specific_data}:
> +
> +\begin{lstlisting}
> +struct virtio_admin_cmd_legacy_dev_cfg_rd_data {
> + u8 offset; /* Starting byte offset of the device configuration region to read */
> +};
> +\end{lstlisting}
> +
> +The driver sets command \field{opcode} to VIRTIO_ADMIN_CMD_LEGACY_DEV_REG_READ.
> +The driver sets valid \field{offset} of the device configurationi region from
> +where to read \field{region_data}.
> +
> +When command completes successfully, \field{command_specific_result}
> +uses following listed structure:
> +
> +\begin{lstlisting}
> +struct virtio_admin_cmd_ld_reg_rd_result {
> + u8 region_data[];
> +};
> +\end{lstlisting}
> +
> +\devicenormative{\paragraph}{Legacy Interfaces: legacy configuration access commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interfaces: legacy configuration access commands}
> +
> +If the group owner device supports legacy registers access for its group members,
> +the device MUST set all corresponding bits for commands
> +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE,
> +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ,
> +VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ in
> +the command result of VIRTIO_ADMIN_CMD_LIST_QUERY in
> +\field{device_admin_cmd_opcodes}.
> +
> +The device MUST encode and decode legacy device specific registers using
> +little-endian format.
> +
> +The device MUST fail VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE and
> +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ commands for the invalid offset
> +which is outside the legacy common configuration register's address range.
> +
> +The device MUST fail VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE,
> +VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ commands for the invalid offset
> +which is outside the legacy device specific register's address range.
> +
> +\drivernormative{\paragraph}{Legacy Interfaces: legacy configuration access commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interfaces: legacy configuration access commands}
> +
> +The driver MUST encode and decode legacy device specific registers using
> +little-endian format.
> +
> +The driver SHOULD send VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE and
> +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ commands with a valid offset which
> +is in the legacy common configuration region address range.
> +
> +The driver SHOULD send commands VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and
> +VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ with a valid offset which is in the legacy
> +device specific configuration region address range.
> diff --git a/admin.tex b/admin.tex
> index fd3b97d..49a0b6f 100644
> --- a/admin.tex
> +++ b/admin.tex
> @@ -117,7 +117,15 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> \hline
> 0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
> \hline
> -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd} \\
> +0x0002 & VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE & Write legacy common configuration region of a member device \\
> +\hline
> +0x0003 & VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ & Read legacy common configuration region of a member device \\
> +\hline
> +0x0004 & VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE & Write legacy device configuration region of a member device \\
> +\hline
> +0x0005 & VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ & Read legacy device configuration region of a member device \\
> +\hline
> +0x0006 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd} \\
> \hline
> 0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure) \\
> \hline
> @@ -286,6 +294,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> supporting multiple group types, the list of supported commands
> might differ between different group types.
>
> +\input{admin-cmds-legacy-access.tex}
> +
> \devicenormative{\subsubsection}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands}
>
> The device MUST validate \field{opcode}, \field{group_type} and
> diff --git a/conformance.tex b/conformance.tex
> index 01ccd69..6bac4b0 100644
> --- a/conformance.tex
> +++ b/conformance.tex
> @@ -260,6 +260,8 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
> \item Section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Legacy Interfaces: A Note on Virtqueue Layout}
> \item Section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Legacy Interfaces: A Note on Virtqueue Endianness}
> \item Section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Message Framing / Legacy Interface: Message Framing}
> +\item Section \ref{devicenormative:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interfaces: legacy configuration access commands}
> +\item Section \ref{drivernormative:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interfaces: legacy configuration access commands}
> \item Section \ref{sec:General Initialization And Device Operation / Device Initialization / Legacy Interface: Device Initialization}
> \item Section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Discovery / Legacy Interfaces: A Note on PCI Device Discovery}
> \item Section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Legacy Interfaces: A Note on PCI Device Layout}
> --
> 2.26.2
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]