RE: [virtio-comment] Re: [PATCH V2 4/6] virtio-pci: implement VIRTIO_F_QUEUE_STATE

[Parav Pandit <parav@nvidia.com>]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Thursday, November 9, 2023 2:12 PM
> 
> On Thu, Nov 09, 2023 at 06:28:17AM +0000, Parav Pandit wrote:
> > > >> The registers are control path in config space, how 400G or 800G affect??
> > > > Because those are the one in practice requires large number of VQs.
> > > >
> > > > You are asking per VQ register commands to modify things
> > > > dynamically via
> > > this one vq at a time, serializing all the operations.
> > > > It does not scale well with high q count.
> > > This is not dynamically, it only happens when SUSPEND and RESUME.
> > > This is the same mechanism how virtio initialize a virtqueue,
> > > working for many years.
> > No. when virtio driver initializes it for the first time, there is no active traffic
> that gets lost.
> > This is because the interface is not yet up and not part of the network yet.
> >
> > The resume must be fast enough, because the remote node is sending
> packets.
> > Hence it is different from driver init time queue enable.
> 
> Maybe but I think not qualitatively different.
> If you care about these things please provide some estimates.
> I just don't see how queue resume writes even with 64k queues will saturate an
> express link.
> 
It is not the bw of the link.
It is how the need for the device to be always read to suspend those many queues through register interface.

> 
> > > >> See the virtio common cfg, you will find the max number of vqs is
> > > >> there, num_queues.
> > > > :)
> > > > Sure. those values at high q count affects.
> > > the driver need to initialize them anyway.
> > That is before the traffic starts from remote end.
> >
> > > >
> > > >>> Device didnât support LM.
> > > >>> Many limitations existed all these years and TC is improving and
> > > >>> expanding
> > > >> them.
> > > >>> So all these years do not matter.
> > > >> Not sure what are you talking about, haven't we initialize the
> > > >> device and vqs in config space for years?????? What's wrong with this
> mechanism?
> > > >> Are you questioning virito-pci fundamentals???
> > > > Donât point to in-efficient past to establish similar in-efficient future.
> > > interesting, you know this is a one-time thing, right?
> > > and you are aware of this has been there for years.
> > > >
> > > >>>>>> Like how to set a queue size and enable it?
> > > >>>>> Those are meant to be used before DRIVER_OK stage as they are
> > > >>>>> init time
> > > >>>> registers.
> > > >>>>> Not to keep abusing them..
> > > >>>> don't you need to set queue_size at the destination side?
> > > >>> No.
> > > >>> But the src/dst does not matter.
> > > >>> Queue_size to be set before DRIVER_OK like rest of the
> > > >>> registers, as all
> > > >> queues must be created before the driver_ok phase.
> > > >>> Queue_reset was last moment exception.
> > > >> create a queue? Nvidia specific?
> > > >>
> > > > Huh. No.
> > > > Do git log and realize what happened with queue_reset.
> > > You didn't answer the question, does the spec even has defined "create a
> vq"?
> >
> > Enabled/created = tomato/tomato when discussing the spec in non-normative
> email conversation.
> > It's irrelevant.
> >
> > All I am saying is, when we know the limitations of the transport and
> > when industry is forwarding to not introduced more and more on-die register
> for once in lifetime work of device migration, we just use the optimal command
> and queue interface that is native to virtio.
> 
> we really do not need to prematurely optimize all things.
> control path is control path it is going to be slow because virtio designed it to be
> slow and drivers don't optimize it.
> Shaving off a microsecond here or there is going to do nothing except increase
> maintainance costs.

Control path post device initialization for large part is through the cvq.