RE: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Parav Pandit <parav@nvidia.com>]

> From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> Sent: Tuesday, June 27, 2023 4:23 AM
> 
> Thanks Parav for pointing it out. We may have some gaps on the case.
> 
> Let me introduce our case, which I think it is simple and should be easy to
> understand.
> 
> First, the user (customer) purchased a bare metal machine.
> 
> ## Bare metal machine
> 
> Let me briefly explain the characteristics of a bare metal machine. It is not a
> virtual machine, it is a physical machine, and the difference between it and a
> general physical machine is that its PCI is connected to a device similar to a DPU.
> This DPU provides devices such as virtio-blk/net to the host through PCI.
> These devices are managed by the vendor, and must be created and purchased
> on the vendor's management platform.
> 
Each of this device can have a unique serial number or id.
Or a parent PF can have the unique serial number.

> ## DPU
> 
> There is a software implementation in the DPU, which will respond to PCI
> operations. But as mentioned above, resources such as network cards must be
> purchased and created before they can exist. So users can create VF, which is
> just a pci-level operation, but there may not be a corresponding backend.
> 
> ## Management Platform
> 
> The creation and configuration of devices is realized on the management
> platform.
> 
> After the user completed the purchase on the management platform (this is an
> independent platform provided by the vendor and has nothing to do with
> virtio), then there will be a corresponding device implementation in the DPU.
> This includes some user configurations, available bandwidth resources and
> other information.
> 
> ## Usage
> 
> Since the user is directly on the HOST, the user can create VMs, passthrough PF
> or VF into the VM. Or users can create a large number of dockers, all of which
> use a separate virtio-net device for performance.
> 
> The reason why users use vf is that we need to use a large number of virtio-net
> devices. This number reaches 1k+.
> 
> Based on this scenario, we need to bind vf to the backend device. Because, we
> cannot automatically complete the creation of the virtio-net backend device
> when the user creates a vf.
> 
Yes, this is common practice for us as well.
A backend device needs to know the unique serial number/id of the virtio device.

> ## Migration
> 
> In addition, let's consider another scenario of migration. If a vm is migrated
> from another host, of course its corresponding virtio device is also migrated to
> the DPU. At this time, our newly created vf can only be used by the vm after it is
> bound to the migrated device. We do not want this vf to be a brand new device.
>
Same for us too.
 
> ## Abstraction
> 
> So, this is how I understand the process of creating vf:
> 
> 1. Create a PCI VF, at this time there may be no backend virtio device, or there
>     is only a default backend. It does not fully meet our expectations.
> 2. Create device or migrate device
> 3. Bind the backend virtio device to the vf
> 
> In most scenarios, the first step may be enough. We can make some fine-tuning
> on this default device, such as modifying its mac. In the future, we can use
> admin queue to modify its msix vector and other configurations.
> 
Yep.

> But we should allow, we bind a backend virtio device to a certain vf. This is
> useful for live migration and virtio devices with special configurations.
>
This is the job of non virtio layer to do the work in backend.
 
> The design of virtio itself is two layers, and virtio should allow switching the
> transport layer by nature. This is our advantage.
> 
> ## About the identity
> 
> In this patch, I used a vendor's id. The purpose of this is that I hope to be
> compatible with all devices. In the network scenario, it is actually possible to
> use a mac.
>
Mac is not good as during migration there is transient where two vfs will have same mac.

So we should use the unique device identifier.

There are few options that I am aware of.

1. Have unique serial number for the PCI PF in the PCI VPD data and VF number to uniquely identify the VF of a PF.
Management system backend will know and able to identify this.
Pros:
This does not need any virtio spec involvement. Can be done today on existing virtio device.
Cons:
There is no single device handle to work with backend, but not a big problem.
If the REST APIs etc of the VPC are crafted properly.

2. Expose unique serial number per each VF using same above PCI VPD data.
This option #2 is good in theory, not efficient to implement as PCI level.
Pros:
a. Unique per device.
b. accessible without binding to the actual driver (vfio vs virtio for different use case)

3. Expose unique serial number/identifier at virtio device level over a virtqueue interface
Pros:
a. achieves the desired scale
b. unique per device
Cons:
a. One needs to bind to the driver to discover the unique device identifier and involve the device reset flow.
Not an optimal solution when dealing with container.

In practice, we found #1 solves most issues.

 
> Perhaps, introduce a standard id for each device/driver, I think this may be more
> in line with the habit of virtio spec.
> 

One way this is solved today for non virtio already is:

when a VF is created a PCI VF device should have the VPD data to contain a unique ID.

> Thanks.
> 
> On Mon, 26 Jun 2023 14:22:10 +0800, Xuan Zhuo
> <xuanzhuo@linux.alibaba.com> wrote:
> > The VFs of the SR-IOV are created by the user inside the guest OS, so
> > the virtio devices don't know about these VFs. Because each VF may be
> > assigned a different role by the user, the virtio device can not choose one VF
> to bind random.
> > So only the user knows how to bind the virtio devices to the VFs.
> > On the other hand, generally the virtio devices are not created by the
> > user inside the guest OS. This requires some management platform to
> participate.
> >
> > So the usage of this command:
> > 1. The user purchases a virtio network card on the management platform,
> >    and sets the ip, queue number, etc. The user obtains the identity of
> >    the network card.
> > 2. The user creates a VF with echo 8 > sriov_numvfs 3. The user binds
> > the net crad to a VF with identity through the command
> >    of the patch
> >
> > Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > ---
> >  admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
> >  1 file changed, 40 insertions(+), 1 deletion(-)
> >
> > diff --git a/admin.tex b/admin.tex
> > index 2efd4d7..64d0667 100644
> > --- a/admin.tex
> > +++ b/admin.tex
> > @@ -115,7 +115,8 @@ \subsection{Group administration
> > commands}\label{sec:Basic Facilities of a Virti  \hline \hline
> >  0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of
> commands supported for this group type    \\
> >  0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of
> commands used for this group type \\
> > -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> > +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one
> group member \\
> > +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> >  \hline
> >  0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a
> different structure)    \\
> >  \hline
> > @@ -429,6 +430,44 @@ \subsection{Group administration
> > commands}\label{sec:Basic Facilities of a Virti  \field{VF Enable}
> > refer to registers within the SR-IOV Extended  Capability as specified by
> \hyperref[intro:PCIe]{[PCIe]}.
> >
> > +\subsubsection{Bind the device for member}
> > +
> > +The VFs of the SR-IOV are created by the user inside the guest OS, so
> > +the virtio devices don't know about these VFs. Because each VF may be
> > +assigned a different role by the user, the virtio device can not choose one VF
> to bind random.
> > +So only the user knows how to bind the virtio devices to the VFs.
> > +On the other hand, generally the virtio devices are not created by
> > +the user inside the guest OS. This requires some management platform to
> participate.
> > +
> > +So we introduce a new admin queue command to bind the VFs and the
> > +virtio devices.
> > +
> > +\begin{lstlisting}
> > +struct virtio_admin_cmd_bind {
> > +    u64 identity;
> > +};
> > +\end{lstlisting}
> > +
> > +The user got the \field{identity} from the management platform, that
> > +is not included by this spec.
> > +
> > +\drivernormative{\paragraph}{Group administration commands}{Basic
> > +Facilities of a Virtio Device / Device groups / Group administration
> > +commands / Bind the device for member}
> > +
> > +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the
> \field{group_member_id} MUST be set.
> > +
> > +The \field{identity} is passed by the user. It is the identity of the
> > +virtio device.
> > +
> > +\devicenormative{\paragraph}{Group administration commands}{Basic
> > +Facilities of a Virtio Device / Device groups / Group administration
> > +commands / Bind the device for member}
> > +
> > +Every device MUST have one unique \field{identity} in the host.
> > +
> > +If the PF device can not find the device by the \field{identity}, the
> > +\field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
> > +
> > +If the device is found by the \field{identity}, the device MUST work
> > +as the device of this group member specified by the
> \field{group_member_id}.
> > +
> >  \section{Administration Virtqueues}\label{sec:Basic Facilities of a
> > Virtio Device / Administration Virtqueues}
> >
> >  An administration virtqueue of an owner device is used to submit
> > --
> > 2.32.0.3.g01195cf9f
> >
> >
> > This publicly archived list offers a means to provide input to the
> > OASIS Virtual I/O Device (VIRTIO) TC.
> >
> > In order to verify user consent to the Feedback License terms and to
> > minimize spam in the list archive, subscription is required before
> > posting.
> >
> > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > List help: virtio-comment-help@lists.oasis-open.org
> > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > Feedback License:
> > https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > List Guidelines:
> > https://www.oasis-open.org/policies-guidelines/mailing-lists
> > Committee: https://www.oasis-open.org/committees/virtio/
> > Join OASIS: https://www.oasis-open.org/join/
> >