Re: [virtio-comment] Re: [PATCH V2 4/6] virtio-pci: implement VIRTIO_F_QUEUE_STATE

["Michael S. Tsirkin" <mst@redhat.com>]

On Thu, Nov 09, 2023 at 09:10:55AM +0000, Parav Pandit wrote:
> 
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Thursday, November 9, 2023 2:12 PM
> > 
> > On Thu, Nov 09, 2023 at 06:28:17AM +0000, Parav Pandit wrote:
> > > > >> The registers are control path in config space, how 400G or 800G affect??
> > > > > Because those are the one in practice requires large number of VQs.
> > > > >
> > > > > You are asking per VQ register commands to modify things
> > > > > dynamically via
> > > > this one vq at a time, serializing all the operations.
> > > > > It does not scale well with high q count.
> > > > This is not dynamically, it only happens when SUSPEND and RESUME.
> > > > This is the same mechanism how virtio initialize a virtqueue,
> > > > working for many years.
> > > No. when virtio driver initializes it for the first time, there is no active traffic
> > that gets lost.
> > > This is because the interface is not yet up and not part of the network yet.
> > >
> > > The resume must be fast enough, because the remote node is sending
> > packets.
> > > Hence it is different from driver init time queue enable.
> > 
> > Maybe but I think not qualitatively different.
> > If you care about these things please provide some estimates.
> > I just don't see how queue resume writes even with 64k queues will saturate an
> > express link.
> > 
> It is not the bw of the link.
> It is how the need for the device to be always read to suspend those many queues through register interface.

read->ready?

If we want to we can solve it btw. Prohibit changing queue select
while suspend is in progress.

But more importantly once Zhu Lingshan stops arguing about suspend bit
he'll hopefully work on transport vq. Then we can have it in config
space and at the same time not use up a register.


> > 
> > > > >> See the virtio common cfg, you will find the max number of vqs is
> > > > >> there, num_queues.
> > > > > :)
> > > > > Sure. those values at high q count affects.
> > > > the driver need to initialize them anyway.
> > > That is before the traffic starts from remote end.
> > >
> > > > >
> > > > >>> Device didnât support LM.
> > > > >>> Many limitations existed all these years and TC is improving and
> > > > >>> expanding
> > > > >> them.
> > > > >>> So all these years do not matter.
> > > > >> Not sure what are you talking about, haven't we initialize the
> > > > >> device and vqs in config space for years?????? What's wrong with this
> > mechanism?
> > > > >> Are you questioning virito-pci fundamentals???
> > > > > Donât point to in-efficient past to establish similar in-efficient future.
> > > > interesting, you know this is a one-time thing, right?
> > > > and you are aware of this has been there for years.
> > > > >
> > > > >>>>>> Like how to set a queue size and enable it?
> > > > >>>>> Those are meant to be used before DRIVER_OK stage as they are
> > > > >>>>> init time
> > > > >>>> registers.
> > > > >>>>> Not to keep abusing them..
> > > > >>>> don't you need to set queue_size at the destination side?
> > > > >>> No.
> > > > >>> But the src/dst does not matter.
> > > > >>> Queue_size to be set before DRIVER_OK like rest of the
> > > > >>> registers, as all
> > > > >> queues must be created before the driver_ok phase.
> > > > >>> Queue_reset was last moment exception.
> > > > >> create a queue? Nvidia specific?
> > > > >>
> > > > > Huh. No.
> > > > > Do git log and realize what happened with queue_reset.
> > > > You didn't answer the question, does the spec even has defined "create a
> > vq"?
> > >
> > > Enabled/created = tomato/tomato when discussing the spec in non-normative
> > email conversation.
> > > It's irrelevant.
> > >
> > > All I am saying is, when we know the limitations of the transport and
> > > when industry is forwarding to not introduced more and more on-die register
> > for once in lifetime work of device migration, we just use the optimal command
> > and queue interface that is native to virtio.
> > 
> > we really do not need to prematurely optimize all things.
> > control path is control path it is going to be slow because virtio designed it to be
> > slow and drivers don't optimize it.
> > Shaving off a microsecond here or there is going to do nothing except increase
> > maintainance costs.
> 
> Control path post device initialization for large part is through the cvq.

Really depends on the device. For virtio net most of the time is
spent on filling up receive queues and sending network announcements
and such.

-- 
MST