[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]
Subject: Re: [PATCH v3 2/3] transport-pci: Introduce legacy registers access commands
On Fri, Jun 02, 2023 at 11:36:03PM +0300, Parav Pandit wrote: > This patch introduces legacy registers access commands for the owner > group member PCI PF to access the legacy registers of the member VFs. > > If in future any SIOV devices to support legacy registers, they > can be easily supported using same commands by using the group > member identifiers of the future SIOV devices. > > More details as overview, motivation, use case are further described > below. > > Usecase: > -------- > 1. A hypervisor/system needs to provide transitional > virtio devices to the guest VM at scale of thousands, > typically, one to eight devices per VM. > > 2. A hypervisor/system needs to provide such devices using a > vendor agnostic driver in the hypervisor system. > > 3. A hypervisor system prefers to have single stack regardless of > virtio device type (net/blk) and be future compatible with a > single vfio stack using SR-IOV or other scalable device > virtualization technology to map PCI devices to the guest VM. > (as transitional or otherwise) > > Motivation/Background: > ---------------------- > The existing virtio transitional PCI device is missing support for > PCI SR-IOV based devices. Currently it does not work beyond > PCI PF, or as software emulated device in reality. Currently it > has below cited system level limitations: > > [a] PCIe spec citation: > VFs do not support I/O Space and thus VF BARs shall not indicate I/O Space. > > [b] cpu arch citiation: > Intel 64 and IA-32 Architectures Software Developerâs Manual: > The processorâs I/O address space is separate and distinct from > the physical-memory address space. The I/O address space consists > of 64K individually addressable 8-bit I/O ports, numbered 0 through FFFFH. > > [c] PCIe spec citation: > If a bridge implements an I/O address range,...I/O address range will be > aligned to a 4 KB boundary. > > Overview: > --------- > Above usecase requirements can be solved by PCI PF group owner accessing > its group member PCI VFs legacy registers using an admin virtqueue of > the group owner PCI PF. > > Two new admin virtqueue commands are added which read/write PCI VF > registers. > > The third command suggested by Jason queries the VF device's driver > notification region. > > Software usage example: > ----------------------- > One way to use and map to the guest VM is by using vfio driver > framework in Linux kernel. > > +----------------------+ > |pci_dev_id = 0x100X | > +---------------|pci_rev_id = 0x0 |-----+ > |vfio device |BAR0 = I/O region | | > | |Other attributes | | > | +----------------------+ | > | | > + +--------------+ +-----------------+ | > | |I/O BAR to AQ | | Other vfio | | > | |rd/wr mapper | | functionalities | | > | +--------------+ +-----------------+ | > | | > +------+-------------------------+-----------+ > | | > | | > | Driver notifications > | | > +----+------------+ +----+------------+ > | +-----+ | | PCI VF device A | > | | AQ |-------------+---->+-------------+ | > | +-----+ | | | | legacy regs | | > | PCI PF device | | | +-------------+ | > +-----------------+ | +-----------------+ > | > | +----+------------+ > | | PCI VF device N | > +---->+-------------+ | > | | legacy regs | | > | +-------------+ | > +-----------------+ > > 2. Virtio pci driver to bind to the listed device id and > use it as native device in the host. > > 3. Use it in a light weight hypervisor to run bare-metal OS. > > Please review. > > Fixes: https://github.com/oasis-tcs/virtio-spec/issues/167 > Signed-off-by: Parav Pandit <parav@nvidia.com> > --- > changelog: > v2->v3: > - adddressed Jason and Michael's comment to split single register > access command to common config and device specific commands. > - dropped the suggetion to introduce enable/disable command as > admin command cap bit already covers it. > v1->v2: > - addressed comments from Michael > - added theory of operation > - grammar corrections > - removed group fields description from individual commands as > it is already present in generic section > - added endianness normative for legacy device registers region > - renamed the file to drop vf and add legacy prefix > > - added overview in commit log > - renamed subsection to reflect command > --- > admin.tex | 12 ++- > transport-pci-legacy-regs.tex | 153 ++++++++++++++++++++++++++++++++++ > transport-pci.tex | 2 + > 3 files changed, 166 insertions(+), 1 deletion(-) > create mode 100644 transport-pci-legacy-regs.tex > > diff --git a/admin.tex b/admin.tex > index e51f9e6..5bcde98 100644 > --- a/admin.tex > +++ b/admin.tex > @@ -117,7 +117,17 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti > \hline > 0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\ > \hline > -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd} \\ > +0x0002 & VIRTIO_ADMIN_CMD_LCC_REG_WRITE & Write legacy common configuration registers of a member device \\ > +\hline > +0x0003 & VIRTIO_ADMIN_CMD_LCC_REG_READ & Read legacy common configuration registers of a member device \\ > +\hline > +0x0004 & VIRTIO_ADMIN_CMD_LD_REG_WRITE & Write legacy device registers of a member device \\ > +\hline > +0x0005 & VIRTIO_ADMIN_CMD_LD_REG_READ & Read legacy device registers of a member device \\ > +\hline > +0x0006 & VIRTIO_ADMIN_CMD_LQ_NOTIFY_QUERY & Read the queue notification offset for legacy interface \\ > +\hline Could you avoid such drastic abbreviation in command names? Standard things like CMD,CFG are ok, but LCC/LD will not ring any bells for anyone, except maybe confusingly make one think of "C Compiler" and "Link eDitor". Let's just LEGACY_COMMON_CFG/LEGACY_DEVICE_CFG? > +0x0007 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd} \\ > \hline > 0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure) \\ > \hline > diff --git a/transport-pci-legacy-regs.tex b/transport-pci-legacy-regs.tex > new file mode 100644 > index 0000000..948ac73 > --- /dev/null > +++ b/transport-pci-legacy-regs.tex > @@ -0,0 +1,153 @@ > +\subsection{Legacy Interfaces: SR-IOV VFs Registers Access}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / Legacy Interfaces: SR-IOV VFs Registers Access} > + > +As described in PCIe base specification \hyperref[intro:PCIe]{[PCIe]} PCI VFs > +do not support the I/O space BAR. A PCI VF hardware device that can support > +legacy interfaces cannot be passthrough to the guest virtual machine > +due to this limitation. To have a transitional virtio device in the guest > +virtual machine, a PCI PF device as a group owner may support accessing its > +group member PCI VF device's legacy registers using an administrative > +virtqueue. This enables minimal intervention in the hypervisor only for > +the purpose of legacy registers access. > + > +A hypervisor software accesses the legacy configuration and device specific > +registers requested by the guest virtual machine using an administrative > +virtqueue. Even though virtqueue driver notifications can be communicated > +through administrative virtqueue, if the PCI PF and VF devices support such > +notifications using a memory-mapped operation, such driver notifications > +are sent using a device defined notification region. > + > +The group owner PCI PF device can optionally enable the driver to access > +its member PCI VFs devices legacy common configuration and device configuration > +registers using an administration virtqueue. Such a PCI PF device supports > +the following commands: > + > +\begin{enumerate} > +\item Legacy Common Configuration Registers Write Command > +\item Legacy common Configuration Registers Read Command > +\item Legacy Device Specific registers Write Command > +\item Legacy Device Specific registers Read Command > +\item Legacy Queue Notify Offset Query Command > +\end{enumerate} > + > +\subsubsection{Legacy Common Configuration Registers Write Command}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / Legacy Interfaces: SR-IOV VFs Registers Access / Legacy Common Configuration Registers Write Command} > + > +The Legacy Common Configuration Registers Write Command follows > +\field{struct virtio_admin_cmd}. > +This command writes legacy common configuration registers of a member VF device. > +The driver sets command \field{opcode} to VIRTIO_ADMIN_CMD_LCC_REG_WRITE. > + > +The command VIRTIO_ADMIN_CMD_LCC_REG_WRITE uses following structure for > +\field{command_specific_data}: > + > +\begin{lstlisting} > +struct virtio_admin_cmd_lcc_reg_wr_data { > + u8 offset; /* Starting byte offset of the common configuration register(s) to write */ > + u8 register[]; > +}; > +\end{lstlisting} > + > +This command does not have any command specific result. > + > +\subsubsection{Legacy Common Configuration Registers Read Command}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / Legacy Interfaces: SR-IOV VFs Registers Access / Legacy Common Configuration Registers Read Command} > + > +The Legacy Common Configuration Registers Read Command follows \field{struct virtio_admin_cmd}. > +This command reads legacy common configuration registers of a member VF device. > +The driver sets command \field{opcode} to VIRTIO_ADMIN_CMD_LCC_REG_READ. > + > +The command VIRTIO_ADMIN_CMD_LCC_REG_READ uses following listed structure for > +\field{command_specific_data}: > + > +\begin{lstlisting} > +struct virtio_admin_cmd_lcc_reg_rd_data { > + u8 offset; /* Starting byte offset of the register to read */ > +}; > +\end{lstlisting} > + > +When command completes successfully, \field{command_specific_result} > +uses following listed structure: > + > +\begin{lstlisting} > +struct virtio_admin_cmd_lcc_reg_rd_result { > + u8 registers[]; > +}; > +\end{lstlisting} > + > + > +\subsubsection{Legacy Device Registers Write Command}\label{sec:Virtio Transport > +Options / Virtio Over PCI Bus / Legacy Interfaces: SR-IOV VFs Registers Access / Legacy Device Registers Write Command} > + > +The Legacy Device Registers Write Command follows \field{struct virtio_admin_cmd}. > +This command writes legacy device registers of a member VF device. > +The driver sets command \field{opcode} to VIRTIO_ADMIN_CMD_LD_REG_WRITE. > + > +The command VIRTIO_ADMIN_CMD_LD_REG_WRITE uses following structure for > +\field{command_specific_data}: > + > +\begin{lstlisting} > +struct virtio_admin_cmd_ld_reg_wr_data { > + u8 offset; /* Starting byte offset of the device register(s) to write */ > + u8 register[]; > +}; > +\end{lstlisting} > + > +This command does not have any command specific result. > + > +\subsubsection{Legacy Device Registers Read Command}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / Legacy Interfaces: SR-IOV VFs Registers Access / Legacy Common Configuration Registers Read Command} > + > +The Legacy Device Registers Read Command follows \field{struct virtio_admin_cmd}. > +This command reads legacy device specific registers of a member VF device. > +The driver sets command \field{opcode} to VIRTIO_ADMIN_CMD_LD_REG_READ. > + > +The command VIRTIO_ADMIN_CMD_LD_REG_READ uses following listed structure for > +\field{command_specific_data}: > + > +\begin{lstlisting} > +struct virtio_admin_cmd_ld_reg_rd_data { > + u8 offset; /* Starting byte offset of the register to read */ > +}; > +\end{lstlisting} > + > +When command completes successfully, \field{command_specific_result} > +uses following listed structure: > + > +\begin{lstlisting} > +struct virtio_admin_cmd_ld_reg_rd_result { > + u8 registers[]; > +}; > +\end{lstlisting} > + > +\subsubsection{Legacy Queue Notify Offset Query Command}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / Legacy Interfaces: SR-IOV VFs Registers Access / Legacy Queue Notify Offset Query Command} > + > +This command returns the notify offset of the member VF for virtqueue > +driver notifications. This command follows \field{struct virtio_admin_cmd}. > +The driver sets command \field{opcode} to VIRTIO_ADMIN_CMD_LQ_NOTIFY_QUERY. > +There is no command specific data for this command. > + > +When command completes successfully, \field{command_specific_result} > +uses following listed structure: > + > +\begin{lstlisting} > +struct virtio_admin_cmd_lq_notify_query_result { > + u8 bar; /* PCI BAR number of the member VF */ > + u8 reserved[7]; > + le64 offset; /* Byte offset within the BAR */ > +}; > +\end{lstlisting} > + > +The driver that may use the driver notifications region of the VF device > +returned in this result likely attain higher performance or the drier may use > +the VIRTIO_ADMIN_CMD_LREG_WRITE command. > + > +\begin{note} > +The device and driver must encode and decode legacy device specific registers > +using little endian format. Per PCI VF device level big endian format support > +is left for the future. > +\end{note} > + > +\begin{note} > +The PCI VF device should not use PCI BAR 0 when it prefers to support > +legacy interface registers access using its group owner PF. This enables > +hypervisor software to operate with least complexities to compose a legacy > +interface I/O space BAR and passthrough other PCI BARs and PCI device > +capabilities to the guest virtual machine without any translation. > +\end{note} > diff --git a/transport-pci.tex b/transport-pci.tex > index a5c6719..72c78f6 100644 > --- a/transport-pci.tex > +++ b/transport-pci.tex > @@ -1212,3 +1212,5 @@ \subsubsection{Driver Handling Interrupts}\label{sec:Virtio Transport Options / > re-examine the configuration space to see what changed. > \end{itemize} > \end{itemize} > + > +\input{transport-pci-legacy-regs.tex} > -- > 2.26.2
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]