RE: [virtio-comment] [PATCH 5/5] virtio-pci: implement VIRTIO_F_QUEUE_STATE

[Parav Pandit <parav@nvidia.com>]

> From: Zhu, Lingshan <lingshan.zhu@intel.com>
> Sent: Tuesday, September 12, 2023 2:33 PM

> admin vq require fixed and dedicated resource to serve the VMs, the question
> still remains, does is scale to server big amount of devices migration? how many
> admin vqs do you need to serve 10 VMs, how many for 100? and so on? How to
> scale?
>
Yes, it scales within the AQ and across multiple AQs.
Please consult your board designers to know such limits for your device.
 
> If one admin vq can serve 100 VMs, can it migrate 1000VMs in reasonable time?
> If not, how many exactly.
> 
Yes, it can serve both 100 and 1000 VMs in reasonable time.

> 
> And register does not need to scale, it resides on the VF and only serve
> the VF.
>
Since its per VF, by nature it is linearly growing entity that the board design needs to support read and write with guaranteed timing.
It clearly scaled poor than queue.
 
> It does not reside on the PF to migrate the VFs.
Hence it does not scale and cannot do parallel operation within the VF, unless each register is replicated.

Using register of a queue for bulk data transfer is solved question when the virtio spec was born.
I donât see a point to discuss it.
Snippet from spec: " As a device can have zero or more virtqueues for bulk data transport"

> VFs config space can use the device dedicated resource like the bandwidth.
>
> for AQ, still you need to reserve resource and how much?
It depends on your board, please consult your board designer to know depending on the implementation.
From spec point of view, it should not be same as any other virtqueue.

> > No. I do not agree. It can fail and very hard for board designers.
> > AQs are more reliable way to transport bulk data in scalable manner for tens
> of member devices.
> Really? How often do you observe virtio config space fail?

On Intel Icelake server we have seen it failing with 128 VFs.
And device needs to do very weird things to support 1000+ VFs forever expanding config space, which is not the topic of this discussion anyway.


> >
> >> Please allow me to provide an extreme example, is one single admin vq
> >> limitless, that can serve hundreds to thousands of VMs migration?
> > It is left to the device implementation. Just like RSS and multi queue support?
> > Is one Q enough for 800Gbps to 10Mbps link?
> > Answer is: Not the scope of specification, spec provide the framework to scale
> this way, but not impose on the device.
> Even if not support RSS or MQ, the device still can work with
> performance overhead, not fail.
>
_work_ is subjective. 
The financial transaction (application) failed. Packeted worked.
LM commands were successful, but it was not timely.

Same same..
 
> Insufficient bandwidth & resource caused live migration fail is totally
> different.
Very abstract point and unrelated to administration commands.

> >
> >> If not, two or
> >> three or what number?
> > It really does not matter. Its wrong point to discuss here.
> > Number of queues and command execution depends on the device
> implementation.
> > A financial transaction application can timeout when a device queuing delay
> for virtio net rx queue is long.
> > And we donât put details about such things in specification.
> > Spec takes the requirements and provides driver device interface to
> implement and scale.
> >
> > I still donât follow the motivation behind the question.
> > Is your question: How many admin queues are needed to migrate N member
> devices? If so, it is implementation specific.
> > It is similar to how such things depend on implementation for 30 virtio device
> types.
> >
> > And if are implying that because it is implementation specific, that is why
> administration queue should not be used, but some configuration register
> should be used.
> > Than you should propose a config register interface to post virtqueue
> descriptors that way for 30 device types!
> if so, leave it as undefined? A potential risk for device implantation?


> Then why must the admin vq?

Because administration commands and admin vq does not impose devices to implement thousands of registers which must have time bound completion guarantee.
The large part of industry including SIOV devices led by Intel and others are moving away from register access mode.

To summarize, administration commands and queue offer following benefits.

1. Ability to do bulk data transfer between driver and device

2. Ability to parallelize the work within driver and within device within single or multiple virtqueues

3. Eliminates implementing PCI read/write MMIO registers which demand low latency response interval

4. Better utilize host cpu as no one needs to poll on the device register for completion

5. Ability to handle variability in command completion by device and ability to notify the driver

If this does not satisfy you, please refer to some of the past email discussions during administration virtuqueue time.