[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]
Subject: Re: [PATCH v7 3/4] admin: Add group member legacy register access commands
On Wed, Jun 28, 2023 at 12:10:49AM +0300, Parav Pandit wrote: > Introduce group member legacy common configuration and legacy device > configuration access read/write commands. > > Group member legacy registers access commands enable group owner driver > software to access legacy registers on behalf of the guest virtual > machine. > > Usecase: > ======== > 1. A hypervisor/system needs to provide transitional > virtio devices to the guest VM at scale of thousands, > typically, one to eight devices per VM. > > 2. A hypervisor/system needs to provide such devices using a > vendor agnostic driver in the hypervisor system. > > 3. A hypervisor system prefers to have single stack regardless of > virtio device type (net/blk) and be future compatible with a > single vfio stack using SR-IOV or other scalable device > virtualization technology to map PCI devices to the guest VM. > (as transitional or otherwise) > > Motivation/Background: > ===================== > The existing virtio transitional PCI device is missing support for > PCI SR-IOV based devices. Currently it does not work beyond > PCI PF, or as software emulated device in reality. Currently it > has below cited system level limitations: > > [a] PCIe spec citation: > VFs do not support I/O Space and thus VF BARs shall not indicate I/O Space. > > [b] cpu arch citiation: > Intel 64 and IA-32 Architectures Software Developerâs Manual: > The processorâs I/O address space is separate and distinct from > the physical-memory address space. The I/O address space consists > of 64K individually addressable 8-bit I/O ports, numbered 0 through FFFFH. > > [c] PCIe spec citation: > If a bridge implements an I/O address range,...I/O address range will be > aligned to a 4 KB boundary. > > Overview: > ========= > Above usecase requirements is solved by PCI PF group owner accessing > its group member PCI VFs legacy registers using the administration > commands of the group owner PCI PF. > > Two types of administration commands are added which read/write PCI VF > registers. > > Software usage example: > ======================= > > 1. One way to use and map to the guest VM is by using vfio driver > framework in Linux kernel. > > +----------------------+ > |pci_dev_id = 0x100X | > +---------------|pci_rev_id = 0x0 |-----+ > |vfio device |BAR0 = I/O region | | > | |Other attributes | | > | +----------------------+ | > | | > + +--------------+ +-----------------+ | > | |I/O BAR to AQ | | Other vfio | | > | |rd/wr mapper& | | functionalities | | > | | forwarder | | | | > | +--------------+ +-----------------+ | > | | > +------+-------------------------+-----------+ > | | > Config region | > access Driver notifications > | | > +----+------------+ +----+------------+ > | +-----+ | | PCI VF device A | > | | AQ |-------------+---->+-------------+ | > | +-----+ | | | | legacy regs | | > | PCI PF device | | | +-------------+ | > +-----------------+ | +-----------------+ > | > | +----+------------+ > | | PCI VF device N | > +---->+-------------+ | > | | legacy regs | | > | +-------------+ | > +-----------------+ > > 2. Continue to use the virtio pci driver to bind to the > listed device id and use it as in the host. > > 3. Use it in a light weight hypervisor to run bare-metal OS. > > Fixes: https://github.com/oasis-tcs/virtio-spec/issues/167 > Signed-off-by: Parav Pandit <parav@nvidia.com> > --- > changelog: > v6->v7: > - changed administrative to administration > - renamed admin-access.tex to admin-interface.tex > - large rewrite ad generic admin commands instead of pci > - added theory of operation section > - added driver notification region query command > v5->v6: > - fixed previous missed abbreviation of LCC and LD > v4->v5: > - split from pci transport specific patch > - split conformance to transport and generic sections > - written the description of the command as generic with member > and group device terminology > - reflected many section names to remove VF > - rename fields from register to region > - avoided abbreviation for legacy, device and config > --- > admin-cmds-legacy-interface.tex | 204 ++++++++++++++++++++++++++++++++ > admin.tex | 14 ++- > conformance.tex | 2 + > 3 files changed, 219 insertions(+), 1 deletion(-) > create mode 100644 admin-cmds-legacy-interface.tex > > diff --git a/admin-cmds-legacy-interface.tex b/admin-cmds-legacy-interface.tex > new file mode 100644 > index 0000000..48d40ee > --- /dev/null > +++ b/admin-cmds-legacy-interface.tex > @@ -0,0 +1,204 @@ > +\subsubsection{Legacy Interfaces}\label{sec:Basic Facilities of a Virtio Device / Device groups / Group > +administration commands / Legacy Interface} > + > +When the member device cannot support certain transport specific > +functionality of a legacy interface, it is difficult to compose a > +transitional group member device. In such scenario, a owner device > +can provide the legacy interface functionality. With such support, > +the member device driver can utilize the group owner device driver legacy > +interface functionality, thereby it can supply either a transitional device > +or a legacy device for the guest virtual machine. For example a PCI VF > +group member device do not support I/O BAR region functionality. For such > +limitation, a PCI PF group owner device support accessing member device's legacy > +configuration region using the administration commands. These administration > +commands are executed by the driver of the owner device when requested by the > +driver of the member device. For example, the member device driver > +intercepts configuration accesses done by the legacy driver and forwards > +such access request to the owner device driver; the owner device driver fulfil > +such requests by executing administration commands on the group owner device. > + > +The group owner device support following administration commands for legacy > +commong configuration and device configuration region access: > + > +\begin{enumerate} > +\item Legacy Common Configuration Region Write Command > +\item Legacy Common Configuration Region Read Command > +\item Legacy Device Configuration Region Write Command > +\item Legacy Device Configuration Region Read Command > +\end{enumerate} > + > +Accessing the member device legacy regions using these commands has the same > +effect as accessing it using the the legacy interface defined by the transport > +of the member device. > + > +\paragraph{Legacy Common Configuration Region Write Command}\label{par:Basic Facilities of a Virtio Device / Device groups / Group > +administration commands / Legacy Interface / Common Configuration Region Write Command} > + > +This command writes in a legacy common configuration region of a group member device. > +This command follows \field{struct virtio_admin_cmd}. > +This command uses following structure for \field{command_specific_data}: > + > +\begin{lstlisting} > +struct virtio_admin_cmd_legacy_common_cfg_wr_data { > + u8 offset; /* Starting byte offset of the common configuration region to write */ > + u8 region_data[]; > +}; > +\end{lstlisting} > + > +The driver sets command \field{opcode} to VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE. > +The driver sets valid \field{offset} and associated \field{region_data} bytes to > +write to the common configuration region. > + > +This command does not have any command specific result. > + > +\paragraph{Legacy Common Configuration Region Read Command}\label{par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interface / Common Configuration Region Read Command} > + > +This command reads from a legacy common configuration region of a group member device. > +This command follows \field{struct virtio_admin_cmd}. > +This command uses following structure for \field{command_specific_data}: > + > +\begin{lstlisting} > +struct virtio_admin_cmd_legacy_common_cfg_rd_data { > + u8 offset; /* Starting byte offset of the common configuration region to read */ > +}; > +\end{lstlisting} > + > +The driver sets command \field{opcode} to VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ. > + > +The driver sets valid \field{offset} of the region from where to read > +\field{region_data}. > + > +When command completes successfully, \field{command_specific_result} > +uses following structure: > + > +\begin{lstlisting} > +struct virtio_admin_cmd_legacy_common_cfg_rd_result { > + u8 region_data[]; > +}; > +\end{lstlisting} > + > +\paragraph{Legacy Device Configuration Region Write Command}\label{par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interface / Device Configuration Region Write Command} > + > +This command writes in a legacy device configuration region of a group member device. > +This command follows \field{struct virtio_admin_cmd}. > +This command uses following structure for \field{command_specific_data}: > + > +\begin{lstlisting} > +struct virtio_admin_cmd_legacy_dev_reg_wr_data { > + u8 offset; /* Starting byte offset of the device configuration region to write */ > + u8 region_data[]; > +}; > +\end{lstlisting} > + > +The driver sets command \field{opcode} to VIRTIO_ADMIN_CMD_LEGACY_DEV_REG_WRITE. > +The driver sets valid \field{offset} and associated \field{region_data} bytes to > +the device configuration region. > + > +This command does not have any command specific result. > + > +\paragraph{Legacy Device Configuration Region Write Command}\label{par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interface / Device Configuration Region Read Command} > + > +This command reads from a legacy device configuration region of a group member device. > +This command follows \field{struct virtio_admin_cmd}. > + > +This command uses following structure for \field{command_specific_data}: > + > +\begin{lstlisting} > +struct virtio_admin_cmd_legacy_dev_cfg_rd_data { > + u8 offset; /* Starting byte offset of the device configuration region to read */ > +}; > +\end{lstlisting} > + > +The driver sets command \field{opcode} to VIRTIO_ADMIN_CMD_LEGACY_DEV_REG_READ. > +The driver sets valid \field{offset} of the device configurationi region from > +where to read \field{region_data}. > + > +When command completes successfully, \field{command_specific_result} > +uses following structure: > + > +\begin{lstlisting} > +struct virtio_admin_cmd_legacy_dev_reg_rd_result { > + u8 region_data[]; > +}; > +\end{lstlisting} > + > +\paragraph{Legacy Driver Notification Region Query}\label{par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interface / Legacy Driver Notifications Region Query} > + > +Even though the driver notifications can be communicated through the > +administration command, if the group owner device or group member device > +supports such notifications using a memory-mapped operation or I/O operation, > +they are sent to the device by accessing this notification region using memory > +or I/O operation. > + > +A group owner device optionally support querying driver notifications region > +using VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY command. > + > +The driver sets command \field{opcode} to VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY. > +This command does not have any command specific data. > + > +When command completes successfully, \field{command_specific_result} > +uses following structure: > + > +\begin{lstlisting} > +struct virtio_admin_cmd_legacy_notify_query_entry { > + u8 region[8]; > +}; This confuses more than it clarifies. Do this: struct virtio_admin_cmd_legacy_notify_query_entry { union { virtio_pci_notify_region region; }; }; > + > +struct virtio_admin_cmd_legacy_notify_query_result { > + struct virtio_virtio_admin_cmd_legacy_notify_query_entry entries[]; > +}; > +\end{lstlisting} > + > +The driver should pick the suitable entry when multiple entries are supplied > +by the device. > + > +Refer to the specific transport section for the definition of the > +\field{region}. Where? How does user know where to look? Add a link. Or preferably I would just include that tex right here to avoid the need to jump back and forth. > + > +\devicenormative{\paragraph}{Legacy Interface}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interface} > + > +If the group owner device supports legacy region access for its group member > +devices, the device MUST set all corresponding bits for commands > +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE, > +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ, > +VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ in > +the command result of VIRTIO_ADMIN_CMD_LIST_QUERY in > +\field{device_admin_cmd_opcodes}. > + > +The device MUST encode and decode legacy device specific registers using > +little-endian format. > + > +The device MUST fail VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE and > +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ commands for the invalid offset > +which is outside the legacy common configuration region's address range. > + > +The device MUST fail VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE, > +VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ commands for the invalid offset > +which is outside the legacy device specific region's address range. > + > +The device SHOULD support VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY command for > +driver notifications. If the group owner device supports driver > +notifications region for its group member devices, the device MUST set > +VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY in the command result of > +VIRTIO_ADMIN_CMD_LIST_QUERY in \field{device_admin_cmd_opcodes}. > + > +When the driver accesses the legacy region of the member device using > +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE, > +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ, > +VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE, VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ > +commands, device MUST function as if they are accessed by the legacy interface > +defined by the transport of the member device. > + > +\drivernormative{\paragraph}{Legacy Interface}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interface} > + > +The driver MUST encode and decode legacy device specific registers using > +little-endian format. > + > +The driver SHOULD send VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE and > +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ commands with a valid offset which > +is in the legacy common configuration region address range. > + > +The driver SHOULD send commands VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and > +VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ with a valid offset which is in the legacy > +device specific configuration region address range. > + > diff --git a/admin.tex b/admin.tex > index fd3b97d..0de26a9 100644 > --- a/admin.tex > +++ b/admin.tex > @@ -117,7 +117,17 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti > \hline > 0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\ > \hline > -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd} \\ > +0x0002 & VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE & Write legacy common configuration region of a member device \\ > +\hline > +0x0003 & VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ & Read legacy common configuration region of a member device \\ > +\hline > +0x0004 & VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE & Write legacy device configuration region of a member device \\ > +\hline > +0x0005 & VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ & Read legacy device configuration region of a member device \\ > +\hline > +0x0006 & VIRTIO_ADMIN_CMD_LEGACY_DEV_NOTIFY_QUERY & Query notification region for a member device \\ > +\hline > +0x0007 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd} \\ > \hline > 0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure) \\ > \hline > @@ -286,6 +296,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti > supporting multiple group types, the list of supported commands > might differ between different group types. > > +\input{admin-cmds-legacy-interface.tex} > + > \devicenormative{\subsubsection}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands} > > The device MUST validate \field{opcode}, \field{group_type} and > diff --git a/conformance.tex b/conformance.tex > index 01ccd69..dc00e84 100644 > --- a/conformance.tex > +++ b/conformance.tex > @@ -260,6 +260,8 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} > \item Section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Legacy Interfaces: A Note on Virtqueue Layout} > \item Section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Legacy Interfaces: A Note on Virtqueue Endianness} > \item Section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Message Framing / Legacy Interface: Message Framing} > +\item Section \ref{devicenormative:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interface} > +\item Section \ref{drivernormative:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interface} > \item Section \ref{sec:General Initialization And Device Operation / Device Initialization / Legacy Interface: Device Initialization} > \item Section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Discovery / Legacy Interfaces: A Note on PCI Device Discovery} > \item Section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Legacy Interfaces: A Note on PCI Device Layout} > -- > 2.26.2
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]