[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]
Subject: Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device
On Tue, 27 Jun 2023 10:57:13 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > > On 6/26/2023 6:50 PM, Xuan Zhuo wrote: > > On Mon, 26 Jun 2023 17:56:01 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > >> > >> On 6/26/2023 5:16 PM, Xuan Zhuo wrote: > >>> On Mon, 26 Jun 2023 16:59:48 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > >>>> On 6/26/2023 4:09 PM, Xuan Zhuo wrote: > >>>>> On Mon, 26 Jun 2023 15:57:33 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > >>>>>> On 6/26/2023 3:08 PM, Xuan Zhuo wrote: > >>>>>>> On Mon, 26 Jun 2023 14:43:17 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > >>>>>>>> On 6/26/2023 2:22 PM, Xuan Zhuo wrote: > >>>>>>>>> The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio > >>>>>>>>> devices don't know about these VFs. Because each VF may be assigned a different role > >>>>>>>>> by the user, the virtio device can not choose one VF to bind random. > >>>>>>>>> So only the user knows how to bind the virtio devices to the VFs. > >>>>>>>>> On the other hand, generally the virtio devices are not created by the user > >>>>>>>>> inside the guest OS. This requires some management platform to participate. > >>>>>>>>> > >>>>>>>>> So the usage of this command: > >>>>>>>>> 1. The user purchases a virtio network card on the management platform, > >>>>>>>>> and sets the ip, queue number, etc. The user obtains the identity of > >>>>>>>>> the network card. > >>>>>>>>> 2. The user creates a VF with echo 8 > sriov_numvfs > >>>>>>>>> 3. The user binds the net crad to a VF with identity through the command > >>>>>>>>> of the patch > >>>>>>>>> > >>>>>>>>> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com> > >>>>>>>>> --- > >>>>>>>>> admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++- > >>>>>>>>> 1 file changed, 40 insertions(+), 1 deletion(-) > >>>>>>>>> > >>>>>>>>> diff --git a/admin.tex b/admin.tex > >>>>>>>>> index 2efd4d7..64d0667 100644 > >>>>>>>>> --- a/admin.tex > >>>>>>>>> +++ b/admin.tex > >>>>>>>>> @@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti > >>>>>>>>> \hline \hline > >>>>>>>>> 0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type \\ > >>>>>>>>> 0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\ > >>>>>>>>> -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd} \\ > >>>>>>>>> +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\ > >>>>>>>>> +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd} \\ > >>>>>>>>> \hline > >>>>>>>>> 0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure) \\ > >>>>>>>>> \hline > >>>>>>>>> @@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti > >>>>>>>>> \field{VF Enable} refer to registers within the SR-IOV Extended > >>>>>>>>> Capability as specified by \hyperref[intro:PCIe]{[PCIe]}. > >>>>>>>>> > >>>>>>>>> +\subsubsection{Bind the device for member} > >>>>>>>>> + > >>>>>>>>> +The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio > >>>>>>>> If the VFs are create in a guest OS, I assume that means the user has > >>>>>>>> passthrough-ed the > >>>>>>>> PF to the guest. For nested, I am not sure whether this is a security > >>>>>>>> issue(affects host pci). > >>>>>>> No care about the passthrough, we always created VFs by the PF. > >>>>>>> > >>>>>>> I should not say "inside the guest OS". I just want to say that the VF is create > >>>>>>> by the user in the OS. The devices does not know about it. > >>>>>> OK, perhaps just say create VFs from a PF in the OS? > >>>>> YES. > >>>>> > >>>>> > >>>>>>>>> +devices don't know about these VFs. Because each VF may be assigned a different role > >>>>>>>>> +by the user, the virtio device can not choose one VF to bind random. > >>>>>>>> I failed to understand this, once a VF is created, it has a personality, > >>>>>>>> e.g., > >>>>>>>> create a virtio-net VF from a virtio-net PF, and PF knows that. > >>>>>>>> > >>>>>>>> I am not familiar with the background, What do you mean by virtio device > >>>>>>>> choose > >>>>>>>> one VF to bind? > >>>>>>> On the cloud, the nic is created by the management platform, the > >>>>>>> user can not create a new nic inside the OS. > >>>>>>> > >>>>>>> So after echo sriov_numvfs, the user just got some VFs, > >>>>>>> there is not backend virtio-net devices. > >>>>>> I think it is not a "user" mange the VFs, the VFs usually provisioned by > >>>>>> the orchestration software and it assign properly selected a VF to a > >>>>>> guest on demands. > >>>>> Yes, but we do not need to care about the guest. Because VF may only be used > >>>>> in host, such as docker. > >>>>> > >>>>> The problem is that the user (you can think of this as the orchestration > >>>>> software) creates some VFs, these are only some PCI devices, which virtio > >>>>> devices will work on these VFs. I think that creating a vf and creating a > >>>>> virtio-net device are two different things. One is done by user in the OS, one > >>>>> is done on the management platform. So we need to bind them together. > >>>> If the VFs are created through sriov_numvfs, once created, the VF device and > >>>> its personality are determined. > >>>> > >>>> PCI spec says: > >>>> All VFs associated with a PF must be the same device type as the PF, > >>>> (e.g., the same network > >>>> device type or the same storage device type.) > >>>> > >>>> So how can the creating process be splitted into separated steps? > >>>> > >>>> Are we discussing something beyond the spec? > >>> NO. > >>> > >>> The device types are same. > >>> > >>> How do we configure the ip, mac, etc of the virtio-net device? In the cloud, > >>> these are managed by the management platform. On the cloud, there is an abstract > >>> object in the backend, which contains things that are generally configured on > >>> the management platform. It is something that users purchase. > >>> Under the virtio standard it is similar to device. > >>> > >>> In my understanding, we just created a pci vf, and virtio works on top of pci, > >>> so there must be two steps here (If I mistake, please point out.). When we create > >>> a vf, it doesn't mean that the backend deivce is ready. Of course, in some > >>> scenarios, we can immediately have a backend default device respond when the > >>> driver probe the vf. But in our scenario, each device is independent. > >> Once a VF is crated, there comes with some default configurations, like > >> MTU and MAC. > >> Do you mean first step creation and second step initialize it? > > Not exactly correct, > > > > The first step is just to create a vf, at this time there can be a default > > virtio-net, it doesn't matter. > > > > In the second step, we can bind a backend device to this vf. > What is "bind a backend device to this vf"? The VF itself is a backend > device, right? I think that the VF is a transport. I want to bind the virtio backend device to this transport. > > > > Not just for initialization for new divice, we also want to support live > > migration. > > > > For example, on the host, we create a vf and passthrough it into a guest os, > > this guest is migrated from another host, and its corresponding network card is > > also migrated to this host. We need to bind this vf to the migrated network > > card. > > > > So just initialization is not enough. > I think you mean to restore the source side NIC configuration? > If so, I think this should be managed in the orchestration layer and > supported by virtio/vdpa APIs, e.g., at least restore common config I will post our case later. Thanks. > > Thanks > > > > Thanks > > > >> If so, current spec only allow the user to config MAC through control vq. > >> vDPA allows to provision a device with proper configuration, maybe that > >> can be the solution? > >> > >> For binding, maybe the orchestration layer manages the pool and it knows > >> how to initialize > >> the device > >> > >> Thanks > >>> Thanks. > >>> > >>>> Thanks > >>>>> Thanks. > >>>>> > >>>>> > >>>>> > >>>>>> So I am confused what the intention of this patch. > >>>>>>> Thanks. > >>>>>>> > >>>>>>> > >>>>>>>>> +So only the user knows how to bind the virtio devices to the VFs. > >>>>>>>>> +On the other hand, generally the virtio devices are not created by the user > >>>>>>>>> +inside the guest OS. This requires some management platform to participate. > >>>>>>>>> + > >>>>>>>>> +So we introduce a new admin queue command to bind the VFs and the virtio > >>>>>>>>> +devices. > >>>>>>>> Sorry, failed to process this. Maybe an orchestration sw layer can help? > >>>>>>>> Provision a device on demands and assign it to a guest? > >>>>>>>> > >>>>>>>> Thanks > >>>>>>>>> + > >>>>>>>>> +\begin{lstlisting} > >>>>>>>>> +struct virtio_admin_cmd_bind { > >>>>>>>>> + u64 identity; > >>>>>>>>> +}; > >>>>>>>>> +\end{lstlisting} > >>>>>>>>> + > >>>>>>>>> +The user got the \field{identity} from the management platform, that is not > >>>>>>>>> +included by this spec. > >>>>>>>>> + > >>>>>>>>> +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member} > >>>>>>>>> + > >>>>>>>>> +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set. > >>>>>>>>> + > >>>>>>>>> +The \field{identity} is passed by the user. It is the identity of the virtio > >>>>>>>>> +device. > >>>>>>>>> + > >>>>>>>>> +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member} > >>>>>>>>> + > >>>>>>>>> +Every device MUST have one unique \field{identity} in the host. > >>>>>>>>> + > >>>>>>>>> +If the PF device can not find the device by the \field{identity}, > >>>>>>>>> +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL. > >>>>>>>>> + > >>>>>>>>> +If the device is found by the \field{identity}, the device MUST work as the > >>>>>>>>> +device of this group member specified by the \field{group_member_id}. > >>>>>>>>> + > >>>>>>>>> \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues} > >>>>>>>>> > >>>>>>>>> An administration virtqueue of an owner device is used to submit > >>>>>>> This publicly archived list offers a means to provide input to the > >>>>>>> OASIS Virtual I/O Device (VIRTIO) TC. > >>>>>>> > >>>>>>> In order to verify user consent to the Feedback License terms and > >>>>>>> to minimize spam in the list archive, subscription is required > >>>>>>> before posting. > >>>>>>> > >>>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org > >>>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > >>>>>>> List help: virtio-comment-help@lists.oasis-open.org > >>>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ > >>>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > >>>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > >>>>>>> Committee: https://www.oasis-open.org/committees/virtio/ > >>>>>>> Join OASIS: https://www.oasis-open.org/join/ > >>>>>>> > >>>>>> This publicly archived list offers a means to provide input to the > >>>>>> OASIS Virtual I/O Device (VIRTIO) TC. > >>>>>> > >>>>>> In order to verify user consent to the Feedback License terms and > >>>>>> to minimize spam in the list archive, subscription is required > >>>>>> before posting. > >>>>>> > >>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org > >>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > >>>>>> List help: virtio-comment-help@lists.oasis-open.org > >>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ > >>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > >>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > >>>>>> Committee: https://www.oasis-open.org/committees/virtio/ > >>>>>> Join OASIS: https://www.oasis-open.org/join/ > >>>>>> > >>>>> --------------------------------------------------------------------- > >>>>> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org > >>>>> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org > >>>>> > >>> This publicly archived list offers a means to provide input to the > >>> OASIS Virtual I/O Device (VIRTIO) TC. > >>> > >>> In order to verify user consent to the Feedback License terms and > >>> to minimize spam in the list archive, subscription is required > >>> before posting. > >>> > >>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org > >>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > >>> List help: virtio-comment-help@lists.oasis-open.org > >>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ > >>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > >>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > >>> Committee: https://www.oasis-open.org/committees/virtio/ > >>> Join OASIS: https://www.oasis-open.org/join/ > >>> > > This publicly archived list offers a means to provide input to the > > OASIS Virtual I/O Device (VIRTIO) TC. > > > > In order to verify user consent to the Feedback License terms and > > to minimize spam in the list archive, subscription is required > > before posting. > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > List help: virtio-comment-help@lists.oasis-open.org > > List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > Committee: https://www.oasis-open.org/committees/virtio/ > > Join OASIS: https://www.oasis-open.org/join/ > > >
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]