RE: [virtio-comment] Re: [PATCH V2 4/6] virtio-pci: implement VIRTIO_F_QUEUE_STATE

[Parav Pandit <parav@nvidia.com>]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Thursday, November 9, 2023 3:23 PM
> To: Parav Pandit <parav@nvidia.com>
> Cc: Zhu, Lingshan <lingshan.zhu@intel.com>; jasowang@redhat.com;
> eperezma@redhat.com; cohuck@redhat.com; stefanha@redhat.com; virtio-
> comment@lists.oasis-open.org
> Subject: Re: [virtio-comment] Re: [PATCH V2 4/6] virtio-pci: implement
> VIRTIO_F_QUEUE_STATE
> 
> On Thu, Nov 09, 2023 at 09:10:55AM +0000, Parav Pandit wrote:
> >
> > > From: Michael S. Tsirkin <mst@redhat.com>
> > > Sent: Thursday, November 9, 2023 2:12 PM
> > >
> > > On Thu, Nov 09, 2023 at 06:28:17AM +0000, Parav Pandit wrote:
> > > > > >> The registers are control path in config space, how 400G or 800G
> affect??
> > > > > > Because those are the one in practice requires large number of VQs.
> > > > > >
> > > > > > You are asking per VQ register commands to modify things
> > > > > > dynamically via
> > > > > this one vq at a time, serializing all the operations.
> > > > > > It does not scale well with high q count.
> > > > > This is not dynamically, it only happens when SUSPEND and RESUME.
> > > > > This is the same mechanism how virtio initialize a virtqueue,
> > > > > working for many years.
> > > > No. when virtio driver initializes it for the first time, there is
> > > > no active traffic
> > > that gets lost.
> > > > This is because the interface is not yet up and not part of the network yet.
> > > >
> > > > The resume must be fast enough, because the remote node is sending
> > > packets.
> > > > Hence it is different from driver init time queue enable.
> > >
> > > Maybe but I think not qualitatively different.
> > > If you care about these things please provide some estimates.
> > > I just don't see how queue resume writes even with 64k queues will
> > > saturate an express link.
> > >
> > It is not the bw of the link.
> > It is how the need for the device to be always read to suspend those many
> queues through register interface.
> 
> read->ready?
> 
> If we want to we can solve it btw. Prohibit changing queue select while suspend
> is in progress.
> 
> But more importantly once Zhu Lingshan stops arguing about suspend bit he'll
> hopefully work on transport vq. Then we can have it in config space and at the
> same time not use up a register.
Transport vq on the owner device for long term does not make sense as hypervisor should not be involved in viewing this content from TDISP time.
So cvq of whatever vq we want to call from the member device will be just perfect.
And it will behave like any other queues.
This will be uniformly available for VF, SIOV, PF devices regardless of TDISP and with TDISP.
This solves both the scale issue and security issue.

Hence 3 solutions,
1. transport vq for VF using owner device for purpose of device migration
2. not using it and inventing another VQ for TDISP in future and for PF, VF, SIOV and TDISP is overkill.

All the operations of the live migration driver like suspend device or others, seems doable just fine using owner device.
What is missing?

> 
> 
> > >
> > > > > >> See the virtio common cfg, you will find the max number of
> > > > > >> vqs is there, num_queues.
> > > > > > :)
> > > > > > Sure. those values at high q count affects.
> > > > > the driver need to initialize them anyway.
> > > > That is before the traffic starts from remote end.
> > > >
> > > > > >
> > > > > >>> Device didnât support LM.
> > > > > >>> Many limitations existed all these years and TC is improving
> > > > > >>> and expanding
> > > > > >> them.
> > > > > >>> So all these years do not matter.
> > > > > >> Not sure what are you talking about, haven't we initialize
> > > > > >> the device and vqs in config space for years?????? What's
> > > > > >> wrong with this
> > > mechanism?
> > > > > >> Are you questioning virito-pci fundamentals???
> > > > > > Donât point to in-efficient past to establish similar in-efficient future.
> > > > > interesting, you know this is a one-time thing, right?
> > > > > and you are aware of this has been there for years.
> > > > > >
> > > > > >>>>>> Like how to set a queue size and enable it?
> > > > > >>>>> Those are meant to be used before DRIVER_OK stage as they
> > > > > >>>>> are init time
> > > > > >>>> registers.
> > > > > >>>>> Not to keep abusing them..
> > > > > >>>> don't you need to set queue_size at the destination side?
> > > > > >>> No.
> > > > > >>> But the src/dst does not matter.
> > > > > >>> Queue_size to be set before DRIVER_OK like rest of the
> > > > > >>> registers, as all
> > > > > >> queues must be created before the driver_ok phase.
> > > > > >>> Queue_reset was last moment exception.
> > > > > >> create a queue? Nvidia specific?
> > > > > >>
> > > > > > Huh. No.
> > > > > > Do git log and realize what happened with queue_reset.
> > > > > You didn't answer the question, does the spec even has defined
> > > > > "create a
> > > vq"?
> > > >
> > > > Enabled/created = tomato/tomato when discussing the spec in
> > > > non-normative
> > > email conversation.
> > > > It's irrelevant.
> > > >
> > > > All I am saying is, when we know the limitations of the transport
> > > > and when industry is forwarding to not introduced more and more
> > > > on-die register
> > > for once in lifetime work of device migration, we just use the
> > > optimal command and queue interface that is native to virtio.
> > >
> > > we really do not need to prematurely optimize all things.
> > > control path is control path it is going to be slow because virtio
> > > designed it to be slow and drivers don't optimize it.
> > > Shaving off a microsecond here or there is going to do nothing
> > > except increase maintainance costs.
> >
> > Control path post device initialization for large part is through the cvq.
> 
> Really depends on the device. For virtio net most of the time is spent on filling
> up receive queues and sending network announcements and such.

Right for the time. But the issue is not about the time spent.
Issue is for the demand of those registers which are of no use when filling large part of filling receive queues.