[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]
Subject: Re: [virtio-comment] About the plan of Admin Queue
On Thu, Jul 27, 2023 at 2:23âPM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote: > > On Thu, 27 Jul 2023 14:17:53 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > > > > > On 7/27/2023 2:09 PM, Xuan Zhuo wrote: > > > On Thu, 27 Jul 2023 11:56:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > >> > > >> On 7/27/2023 10:30 AM, Xuan Zhuo wrote: > > >>> On Mon, 3 Jul 2023 12:29:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > >>>> On 6/30/2023 7:35 PM, Parav Pandit wrote: > > >>>>>> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis- > > >>>>>> open.org> On Behalf Of Zhu, Lingshan > > >>>>>> Sent: Friday, June 30, 2023 6:33 AM > > >>>>>> > > >>>>>>>>> Can we let the DPU notify the driver to create a new devicer from the > > >>>>>> backend? > > >>>>> Yes, why not. > > >>>>> > > >>>>>>>>> The key point is who want to create a new device. > > >>>>>>>> DPU can come with a certain number of pre-created ADIs, just make > > >>>>>>>> sure the orchestration SW is aware of their device IDs. > > >>>>>>>> > > >>>>> Cloud often need these devices to be created dynamically, many a time after the host OS is booted. > > >>>>> To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host. > > >>>>> Xuan partly explained it. > > >>>>> > > >>>>>>>> If you want the DPU randomly create ADIs and notify the driver, I > > >>>>>>>> think we need interrupt, e.g., re-use config interrupt. But why DPU > > >>>>>>>> wants to create and hot plug in a device to a guest? > > >>>>>>>> Shall the host handle that or DPU pre-create then expose to baremteal > > >>>>>>>> machines? > > >>>>>>> In your scenario, the supervisor is on the os, which controls the DPU > > >>>>>>> to create new devices. > > >>>>>>> > > >>>>>>> In the cloud scenario, the vendor manager is in the DPU, and the > > >>>>>>> entire host is for users. Of course, there are situations where the > > >>>>>>> vendor manager are in the HOST. But for bare metal machines, the host > > >>>>>>> belongs to the customer, the vendor manager is only in the DPU. > > >>>>>>> > > >>>>>>> So when the customers buy a new nic for the host, the vendor manager > > >>>>>>> will plug a device to the host from the DPU. > > >>>>>> I understand once a customer orders a new NIC, you wants to present the NIC > > >>>>>> to the host. > > >>>>>> However you only owns the DPU and the customer owns the host, that means > > >>>>>> this creation and hot plug must be transparent to the host and there may not be > > >>>>>> a host driver help handling an interrupt/probe. > > >>>>>> > > >>>>> That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model. > > >>>>> > > >>>>>> However this is not PCI which has a tree/switch and can enumerate devices to > > >>>>>> the host by spanning the device across the PCI hierarchy. > > >>>>>> > > >>>>> Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it. > > >>>>> > > >>>>>> To address an ADI, there is only a device_id. > > >>>>>> > > >>>>> SIOV device must have a unique identifier at PCI bus level for sure. > > >>>>> I cannot speak more about it in this forum due to other logistics issue. > > >>>>> But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about. > > >>>>> > > >>>>>> So, do you mind share how your DPU offload the device model? What kind of > > >>>>>> device your DPU provide to the host? Lets see whether DPU can mediate this by > > >>>>>> its own? > > >>>>>> > > >>>>> It is a virtio nic, blk and other virtio devices for us. > > >>>>> A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries. > > >>>>> There is no mediation per say here, it is just a dpu based SIOV device like a regular PF. > > >>>>> > > >>>>> For non virtio DPU device, I implemented them in Linux for dpus 2 years ago. > > >>>>> You might find a Linux reference model useful at [1]. > > >>>>> A usage model already exists in one OS and in use for non virtio devices. > > >>>>> This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist. > > >>>>> > > >>>>> For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own. > > >>>>> > > >>>>> [1] https://github.com/Mellanox/scalablefunctions/wiki > > >>>> well I see SF facing the similar challenge, I can add a command for the > > >>>> driver to query all existing SIOV ADIs of a device, > > >>>> and the device return ADIs id and status. Looks good? and work for you > > >>>> @Xuan? > > >>> Could I have your plan for this? > > >>> > > >>> If you do not mind, I'd like to add a command to query VF's info. Such > > >>> as mac, ip, etc. > > >> I think the query commands for SIOV is a little more complex, e.g., > > >> need to report device type and its scale(e.g., features, mq). > > >> There can be thousands of SIOV ADIs and we don't want output flood. > > >> > > >> We have discussed implementation a config interrupt to report new > > >> created / deleted > > >> ADIs on the DPU side, therefore there must be a cap contains related > > >> information, > > >> my rough approach of the process is: > > >> 1) a cap contains the total number of existing ADIs and the max dev id > > >> 2) driver queries detailed information of a certain ADI or a bunch of > > >> ADIs in a [dev_id....dev_id2] range. > > > Yes, Admin Queue can obtain the info of the specific one or more devices. > > > > > >> I am not sure whether a NIC stores its IP > > > > > > IP is the other topic. I want the Admin Queue manage the switch. > > > So the switch know about the IP of every device, and the > > > Admin Queue will has the ability to config the IP of the device inside the > > > switch. > > DPU onboard switch? OVS? Does it beyond virtio spec? > > YES. Adding Washizu. We can have a switch/dpa defined in the networking device for sure. > > For SIOV, I think this is MUST. A learning bridge would be fine as a starter. It's better not to couple new scalable capability with any device specific features. > Maybe you have one simple implementation. > But you have to solve the IP steering. So admin queue should has the ability > to config the IP steering. I think not. Those L2/LN tables/filters are networking specific. Control virtqueue is better than admin virtqueue here. Thanks > > Thanks. > > > > > > > > > > Thanks. > > > > > > > > >> Thanks > > >>> Thanks. > > >>> > > >>> > > >>> > > >>>> Thanks > > >>>> > > >>> This publicly archived list offers a means to provide input to the > > >>> OASIS Virtual I/O Device (VIRTIO) TC. > > >>> > > >>> In order to verify user consent to the Feedback License terms and > > >>> to minimize spam in the list archive, subscription is required > > >>> before posting. > > >>> > > >>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > >>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > >>> List help: virtio-comment-help@lists.oasis-open.org > > >>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > >>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > >>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > >>> Committee: https://www.oasis-open.org/committees/virtio/ > > >>> Join OASIS: https://www.oasis-open.org/join/ > > >>> > > > > > > This publicly archived list offers a means to provide input to the > > OASIS Virtual I/O Device (VIRTIO) TC. > > > > In order to verify user consent to the Feedback License terms and > > to minimize spam in the list archive, subscription is required > > before posting. > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > List help: virtio-comment-help@lists.oasis-open.org > > List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > Committee: https://www.oasis-open.org/committees/virtio/ > > Join OASIS: https://www.oasis-open.org/join/ > > > > This publicly archived list offers a means to provide input to the > OASIS Virtual I/O Device (VIRTIO) TC. > > In order to verify user consent to the Feedback License terms and > to minimize spam in the list archive, subscription is required > before posting. > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > List help: virtio-comment-help@lists.oasis-open.org > List archive: https://lists.oasis-open.org/archives/virtio-comment/ > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > Committee: https://www.oasis-open.org/committees/virtio/ > Join OASIS: https://www.oasis-open.org/join/ >
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]