[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]
Subject: Re: [virtio-comment] About the plan of Admin Queue
On Thu, 27 Jul 2023 16:28:07 +0800, Jason Wang <jasowang@redhat.com> wrote: > On Thu, Jul 27, 2023 at 4:20âPM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote: > > > > On Thu, 27 Jul 2023 16:03:56 +0800, Jason Wang <jasowang@redhat.com> wrote: > > > On Thu, Jul 27, 2023 at 2:23âPM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote: > > > > > > > > On Thu, 27 Jul 2023 14:17:53 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > > > > > > > > > > > > > > On 7/27/2023 2:09 PM, Xuan Zhuo wrote: > > > > > > On Thu, 27 Jul 2023 11:56:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > > > > >> > > > > > >> On 7/27/2023 10:30 AM, Xuan Zhuo wrote: > > > > > >>> On Mon, 3 Jul 2023 12:29:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > > > > >>>> On 6/30/2023 7:35 PM, Parav Pandit wrote: > > > > > >>>>>> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis- > > > > > >>>>>> open.org> On Behalf Of Zhu, Lingshan > > > > > >>>>>> Sent: Friday, June 30, 2023 6:33 AM > > > > > >>>>>> > > > > > >>>>>>>>> Can we let the DPU notify the driver to create a new devicer from the > > > > > >>>>>> backend? > > > > > >>>>> Yes, why not. > > > > > >>>>> > > > > > >>>>>>>>> The key point is who want to create a new device. > > > > > >>>>>>>> DPU can come with a certain number of pre-created ADIs, just make > > > > > >>>>>>>> sure the orchestration SW is aware of their device IDs. > > > > > >>>>>>>> > > > > > >>>>> Cloud often need these devices to be created dynamically, many a time after the host OS is booted. > > > > > >>>>> To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host. > > > > > >>>>> Xuan partly explained it. > > > > > >>>>> > > > > > >>>>>>>> If you want the DPU randomly create ADIs and notify the driver, I > > > > > >>>>>>>> think we need interrupt, e.g., re-use config interrupt. But why DPU > > > > > >>>>>>>> wants to create and hot plug in a device to a guest? > > > > > >>>>>>>> Shall the host handle that or DPU pre-create then expose to baremteal > > > > > >>>>>>>> machines? > > > > > >>>>>>> In your scenario, the supervisor is on the os, which controls the DPU > > > > > >>>>>>> to create new devices. > > > > > >>>>>>> > > > > > >>>>>>> In the cloud scenario, the vendor manager is in the DPU, and the > > > > > >>>>>>> entire host is for users. Of course, there are situations where the > > > > > >>>>>>> vendor manager are in the HOST. But for bare metal machines, the host > > > > > >>>>>>> belongs to the customer, the vendor manager is only in the DPU. > > > > > >>>>>>> > > > > > >>>>>>> So when the customers buy a new nic for the host, the vendor manager > > > > > >>>>>>> will plug a device to the host from the DPU. > > > > > >>>>>> I understand once a customer orders a new NIC, you wants to present the NIC > > > > > >>>>>> to the host. > > > > > >>>>>> However you only owns the DPU and the customer owns the host, that means > > > > > >>>>>> this creation and hot plug must be transparent to the host and there may not be > > > > > >>>>>> a host driver help handling an interrupt/probe. > > > > > >>>>>> > > > > > >>>>> That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model. > > > > > >>>>> > > > > > >>>>>> However this is not PCI which has a tree/switch and can enumerate devices to > > > > > >>>>>> the host by spanning the device across the PCI hierarchy. > > > > > >>>>>> > > > > > >>>>> Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it. > > > > > >>>>> > > > > > >>>>>> To address an ADI, there is only a device_id. > > > > > >>>>>> > > > > > >>>>> SIOV device must have a unique identifier at PCI bus level for sure. > > > > > >>>>> I cannot speak more about it in this forum due to other logistics issue. > > > > > >>>>> But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about. > > > > > >>>>> > > > > > >>>>>> So, do you mind share how your DPU offload the device model? What kind of > > > > > >>>>>> device your DPU provide to the host? Lets see whether DPU can mediate this by > > > > > >>>>>> its own? > > > > > >>>>>> > > > > > >>>>> It is a virtio nic, blk and other virtio devices for us. > > > > > >>>>> A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries. > > > > > >>>>> There is no mediation per say here, it is just a dpu based SIOV device like a regular PF. > > > > > >>>>> > > > > > >>>>> For non virtio DPU device, I implemented them in Linux for dpus 2 years ago. > > > > > >>>>> You might find a Linux reference model useful at [1]. > > > > > >>>>> A usage model already exists in one OS and in use for non virtio devices. > > > > > >>>>> This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist. > > > > > >>>>> > > > > > >>>>> For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own. > > > > > >>>>> > > > > > >>>>> [1] https://github.com/Mellanox/scalablefunctions/wiki > > > > > >>>> well I see SF facing the similar challenge, I can add a command for the > > > > > >>>> driver to query all existing SIOV ADIs of a device, > > > > > >>>> and the device return ADIs id and status. Looks good? and work for you > > > > > >>>> @Xuan? > > > > > >>> Could I have your plan for this? > > > > > >>> > > > > > >>> If you do not mind, I'd like to add a command to query VF's info. Such > > > > > >>> as mac, ip, etc. > > > > > >> I think the query commands for SIOV is a little more complex, e.g., > > > > > >> need to report device type and its scale(e.g., features, mq). > > > > > >> There can be thousands of SIOV ADIs and we don't want output flood. > > > > > >> > > > > > >> We have discussed implementation a config interrupt to report new > > > > > >> created / deleted > > > > > >> ADIs on the DPU side, therefore there must be a cap contains related > > > > > >> information, > > > > > >> my rough approach of the process is: > > > > > >> 1) a cap contains the total number of existing ADIs and the max dev id > > > > > >> 2) driver queries detailed information of a certain ADI or a bunch of > > > > > >> ADIs in a [dev_id....dev_id2] range. > > > > > > Yes, Admin Queue can obtain the info of the specific one or more devices. > > > > > > > > > > > >> I am not sure whether a NIC stores its IP > > > > > > > > > > > > IP is the other topic. I want the Admin Queue manage the switch. > > > > > > So the switch know about the IP of every device, and the > > > > > > Admin Queue will has the ability to config the IP of the device inside the > > > > > > switch. > > > > > DPU onboard switch? OVS? Does it beyond virtio spec? > > > > > > > > YES. > > > > > > Adding Washizu. > > > > > > We can have a switch/dpa defined in the networking device for sure. > > > > Yes, I think we should introduce that for the sr-iov. Or for other. > > This should be a general one as a switch should be transport independent. I agree. > > > > > I would like to know who is doing this? > > Washizu, could you confirm if you want to do this or not? > > > > > Another question, @Jason are you referring to a new device type or a > > new virtio-net feature. > > Extending virtio-net should be fine, did you see any issues for this? You kwnow the packet will be steered to different virtio-net devices. So I think this is odd if the feature is the extending of virtio-net. How do you think about this? > > > > > > > > > > > > > > For SIOV, I think this is MUST. > > > > > > A learning bridge would be fine as a starter. It's better not to > > > couple new scalable capability with any device specific features. > > > > > > > Maybe you have one simple implementation. > > > > But you have to solve the IP steering. So admin queue should has the ability > > > > to config the IP steering. > > > > > > I think not. Those L2/LN tables/filters are networking specific. > > > > Let us assume that there is a switch/bridge firstly. > > The VFs may be passed to different VMs. > > > > I also think this is the networking specific. But I want to config > > the ip for every vf from the pf. > > What do you mean by ip here (e.g who is the user for this ip?) 1. the admin queue config the ip for the vf to the switch 2. the vf get the ip by dhcp from the switch. (or other way) Here we config an ip for the vf, and we also limit that just this vf can use the ip for security. Other vf can not use the ip configured by self. > > > Because the user of the vf may be unreliable. > > We need a manager to config the ip for every vf. > > Did you mean you're using a tunnel or not? No. Multiple VMs on a host may be used by different users. Some users are not trustworthy. We want to prevent a user from maliciously using ip > > > > > > > > Control virtqueue is better than admin virtqueue here. > > > > by cq? > > > > What case? > > We've already used control virtqueue for steering. > More details? You config the steering rule to other virtio-net device by the cq of one virtio-net? Thanks. > Thanks > > > > > Thanks. > > > > > > > > Thanks > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > > > > >> Thanks > > > > > >>> Thanks. > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>>> Thanks > > > > > >>>> > > > > > >>> This publicly archived list offers a means to provide input to the > > > > > >>> OASIS Virtual I/O Device (VIRTIO) TC. > > > > > >>> > > > > > >>> In order to verify user consent to the Feedback License terms and > > > > > >>> to minimize spam in the list archive, subscription is required > > > > > >>> before posting. > > > > > >>> > > > > > >>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > > > > >>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > > > > >>> List help: virtio-comment-help@lists.oasis-open.org > > > > > >>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > > > > >>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > > > > >>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > > > > >>> Committee: https://www.oasis-open.org/committees/virtio/ > > > > > >>> Join OASIS: https://www.oasis-open.org/join/ > > > > > >>> > > > > > > > > > > > > > > > This publicly archived list offers a means to provide input to the > > > > > OASIS Virtual I/O Device (VIRTIO) TC. > > > > > > > > > > In order to verify user consent to the Feedback License terms and > > > > > to minimize spam in the list archive, subscription is required > > > > > before posting. > > > > > > > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > > > > List help: virtio-comment-help@lists.oasis-open.org > > > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > > > > Committee: https://www.oasis-open.org/committees/virtio/ > > > > > Join OASIS: https://www.oasis-open.org/join/ > > > > > > > > > > > > > This publicly archived list offers a means to provide input to the > > > > OASIS Virtual I/O Device (VIRTIO) TC. > > > > > > > > In order to verify user consent to the Feedback License terms and > > > > to minimize spam in the list archive, subscription is required > > > > before posting. > > > > > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > > > List help: virtio-comment-help@lists.oasis-open.org > > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > > > Committee: https://www.oasis-open.org/committees/virtio/ > > > > Join OASIS: https://www.oasis-open.org/join/ > > > > > > > > > >
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]