Re: [virtio-comment] About the plan of Admin Queue

["Zhu, Lingshan" <lingshan.zhu@intel.com>]

On 6/30/2023 7:35 PM, Parav Pandit wrote:
> > From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
> > open.org> On Behalf Of Zhu, Lingshan
> > Sent: Friday, June 30, 2023 6:33 AM
>
> > > > > Can we let the DPU notify the driver to create a new devicer from the
> > backend?
> Yes, why not.
>
> > > > > The key point is who want to create a new device.
> > > > DPU can come with a certain number of pre-created ADIs, just make
> > > > sure the orchestration SW is aware of their device IDs.
> Cloud often need these devices to be created dynamically, many a time after the host OS is booted.
> To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host.
> Xuan partly explained it.
>
> > > > If you want the DPU randomly create ADIs and notify the driver, I
> > > > think we need interrupt, e.g., re-use config interrupt. But why DPU
> > > > wants to create and hot plug in a device to a guest?
> > > > Shall the host handle that or DPU pre-create then expose to baremteal
> > > > machines?
> > > In your scenario, the supervisor is on the os, which controls the DPU
> > > to create new devices.
> > >
> > > In the cloud scenario, the vendor manager is in the DPU, and the
> > > entire host is for users. Of course, there are situations where the
> > > vendor manager are in the HOST. But for bare metal machines, the host
> > > belongs to the customer, the vendor manager is only in the DPU.
> > >
> > > So when the customers buy a new nic for the host, the vendor manager
> > > will plug a device to the host from the DPU.
> > I understand once a customer orders a new NIC, you wants to present the NIC
> > to the host.
> > However you only owns the DPU and the customer owns the host, that means
> > this creation and hot plug must be transparent to the host and there may not be
> > a host driver help handling an interrupt/probe.
> That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
>
> > However this is not PCI which has a tree/switch and can enumerate devices to
> > the host by spanning the device across the PCI hierarchy.
> Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
>
> > To address an ADI, there is only a device_id.
> SIOV device must have a unique identifier at PCI bus level for sure.
> I cannot speak more about it in this forum due to other logistics issue.
> But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.
>   
> > So, do you mind share how your DPU offload the device model? What kind of
> > device your DPU provide to the host? Lets see whether DPU can mediate this by
> > its own?
> It is a virtio nic, blk and other virtio devices for us.
> A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries.
> There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.
>
> For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
> You might find a Linux reference model useful at [1].
> A usage model already exists in one OS and in use for non virtio devices.
> This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.
>
> For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
>
> [1] https://github.com/Mellanox/scalablefunctions/wiki
well I see SF facing the similar challenge, I can add a command for the 
driver to query all existing SIOV ADIs of a device,
and the device return ADIs id and status. Looks good? and work for you 
@Xuan?

Thanks