Re: [virtio-comment] [PATCH 5/5] virtio-pci: implement VIRTIO_F_QUEUE_STATE

["Zhu, Lingshan" <lingshan.zhu@intel.com>]

On 9/12/2023 5:21 PM, Parav Pandit wrote:
> > From: Zhu, Lingshan <lingshan.zhu@intel.com>
> > Sent: Tuesday, September 12, 2023 2:33 PM
> > admin vq require fixed and dedicated resource to serve the VMs, the question
> > still remains, does is scale to server big amount of devices migration? how many
> > admin vqs do you need to serve 10 VMs, how many for 100? and so on? How to
> > scale?
> Yes, it scales within the AQ and across multiple AQs.
> Please consult your board designers to know such limits for your device.
scales require multiple AQs, then how many should a vendor provide for the
worst case?

I am boring for the same repeating questions.
>   
> > If one admin vq can serve 100 VMs, can it migrate 1000VMs in reasonable time?
> > If not, how many exactly.
> Yes, it can serve both 100 and 1000 VMs in reasonable time.
I am not sure, the aq is limitless? Can serve thousands of VMs
in a reasonable time? Like in 300ms?

If you say, that require multiple AQ, then how many should a vendor provide?

Don't say the board designer own the risks.
> > And register does not need to scale, it resides on the VF and only serve
> > the VF.
> Since its per VF, by nature it is linearly growing entity that the board design needs to support read and write with guaranteed timing.
> It clearly scaled poor than queue.
Please read my series. For example, we introduce a new bit SUSPEND in 
the \field{device status}, any scalability issues here?
>   
> > It does not reside on the PF to migrate the VFs.
> Hence it does not scale and cannot do parallel operation within the VF, unless each register is replicated.
Why its not scale? It is a per device facility.
Why do you need parallel operation against the LM facility?
That doesn't make a lot of sense.
> Using register of a queue for bulk data transfer is solved question when the virtio spec was born.
> I donât see a point to discuss it.
> Snippet from spec: " As a device can have zero or more virtqueues for bulk data transport"
Where do you see the series intends to transfer bulk data through registers?
> > VFs config space can use the device dedicated resource like the bandwidth.
> >
> > for AQ, still you need to reserve resource and how much?
> It depends on your board, please consult your board designer to know depending on the implementation.
>  From spec point of view, it should not be same as any other virtqueue.
so the vendor own the risk to implement AQ LM? Why they have to?
> > > No. I do not agree. It can fail and very hard for board designers.
> > > AQs are more reliable way to transport bulk data in scalable manner for tens
> > of member devices.
> > Really? How often do you observe virtio config space fail?
> On Intel Icelake server we have seen it failing with 128 VFs.
> And device needs to do very weird things to support 1000+ VFs forever expanding config space, which is not the topic of this discussion anyway.
That is your setup problem.
> > > > Please allow me to provide an extreme example, is one single admin vq
> > > > limitless, that can serve hundreds to thousands of VMs migration?
> > > It is left to the device implementation. Just like RSS and multi queue support?
> > > Is one Q enough for 800Gbps to 10Mbps link?
> > > Answer is: Not the scope of specification, spec provide the framework to scale
> > this way, but not impose on the device.
> > Even if not support RSS or MQ, the device still can work with
> > performance overhead, not fail.
> _work_ is subjective.
> The financial transaction (application) failed. Packeted worked.
> LM commands were successful, but it was not timely.
>
> Same same..
>   
> > Insufficient bandwidth & resource caused live migration fail is totally
> > different.
> Very abstract point and unrelated to administration commands.
It is your design facing the problem.
> > > > If not, two or
> > > > three or what number?
> > > It really does not matter. Its wrong point to discuss here.
> > > Number of queues and command execution depends on the device
> > implementation.
> > > A financial transaction application can timeout when a device queuing delay
> > for virtio net rx queue is long.
> > > And we donât put details about such things in specification.
> > > Spec takes the requirements and provides driver device interface to
> > implement and scale.
> > > I still donât follow the motivation behind the question.
> > > Is your question: How many admin queues are needed to migrate N member
> > devices? If so, it is implementation specific.
> > > It is similar to how such things depend on implementation for 30 virtio device
> > types.
> > > And if are implying that because it is implementation specific, that is why
> > administration queue should not be used, but some configuration register
> > should be used.
> > > Than you should propose a config register interface to post virtqueue
> > descriptors that way for 30 device types!
> > if so, leave it as undefined? A potential risk for device implantation?
>
> > Then why must the admin vq?
> Because administration commands and admin vq does not impose devices to implement thousands of registers which must have time bound completion guarantee.
> The large part of industry including SIOV devices led by Intel and others are moving away from register access mode.
>
> To summarize, administration commands and queue offer following benefits.
>
> 1. Ability to do bulk data transfer between driver and device
>
> 2. Ability to parallelize the work within driver and within device within single or multiple virtqueues
>
> 3. Eliminates implementing PCI read/write MMIO registers which demand low latency response interval
>
> 4. Better utilize host cpu as no one needs to poll on the device register for completion
>
> 5. Ability to handle variability in command completion by device and ability to notify the driver
>
> If this does not satisfy you, please refer to some of the past email discussions during administration virtuqueue time.
I think you mixed up the facility and the implementation in my series, 
please read.