RE: [virtio-comment] [PATCH 5/5] virtio-pci: implement VIRTIO_F_QUEUE_STATE

[Parav Pandit <parav@nvidia.com>]

> From: Zhu, Lingshan <lingshan.zhu@intel.com>
> Sent: Tuesday, September 12, 2023 6:33 PM
> 
> On 9/12/2023 5:21 PM, Parav Pandit wrote:
> >> From: Zhu, Lingshan <lingshan.zhu@intel.com>
> >> Sent: Tuesday, September 12, 2023 2:33 PM admin vq require fixed and
> >> dedicated resource to serve the VMs, the question still remains, does
> >> is scale to server big amount of devices migration? how many admin
> >> vqs do you need to serve 10 VMs, how many for 100? and so on? How to
> >> scale?
> >>
> > Yes, it scales within the AQ and across multiple AQs.
> > Please consult your board designers to know such limits for your device.
> scales require multiple AQs, then how many should a vendor provide for the
> worst case?
> 
> I am boring for the same repeating questions.
I said it scales, within the AQ. (and across AQs).
I have answered enough times, so I will stop on same repeated question.
Your repeated question is not helping anyone as it is not in the scope of virtio.

If you think it is, please get it written first for RSS and MQ in net section and post for review.

> >
> >> If one admin vq can serve 100 VMs, can it migrate 1000VMs in reasonable
> time?
> >> If not, how many exactly.
> >>
> > Yes, it can serve both 100 and 1000 VMs in reasonable time.
> I am not sure, the aq is limitless? Can serve thousands of VMs in a reasonable
> time? Like in 300ms?
> 
Yes.

> If you say, that require multiple AQ, then how many should a vendor provide?
> 
I didnât say multiple AQs must be used.
It is same as NIC RQs.

> Don't say the board designer own the risks.

> >
> >> And register does not need to scale, it resides on the VF and only
> >> serve the VF.
> >>
> > Since its per VF, by nature it is linearly growing entity that the board design
> needs to support read and write with guaranteed timing.
> > It clearly scaled poor than queue.
> Please read my series. For example, we introduce a new bit SUSPEND in the
> \field{device status}, any scalability issues here?
That must behave like queue_reset, (it must get acknowledged from the device) that it is suspended.
And that brings the scale issue.
On top of that once the device is SUSPENDED, it cannot accept some other RESET_VQ command.

> >
> >> It does not reside on the PF to migrate the VFs.
> > Hence it does not scale and cannot do parallel operation within the VF, unless
> each register is replicated.
> Why its not scale? It is a per device facility.
Because the device needs to answer per device through some large scale memory to fit in a response time.

> Why do you need parallel operation against the LM facility?
Because your downtime was 300msec for 1000 VMs.

> That doesn't make a lot of sense.
> >
> > Using register of a queue for bulk data transfer is solved question when the
> virtio spec was born.
> > I donât see a point to discuss it.
> > Snippet from spec: " As a device can have zero or more virtqueues for bulk
> data transport"
> Where do you see the series intends to transfer bulk data through registers?
> >
> >> VFs config space can use the device dedicated resource like the bandwidth.
> >>
> >> for AQ, still you need to reserve resource and how much?
> > It depends on your board, please consult your board designer to know
> depending on the implementation.
> >  From spec point of view, it should not be same as any other virtqueue.
> so the vendor own the risk to implement AQ LM? Why they have to?
> >>> No. I do not agree. It can fail and very hard for board designers.
> >>> AQs are more reliable way to transport bulk data in scalable manner
> >>> for tens
> >> of member devices.
> >> Really? How often do you observe virtio config space fail?
> > On Intel Icelake server we have seen it failing with 128 VFs.
> > And device needs to do very weird things to support 1000+ VFs forever
> expanding config space, which is not the topic of this discussion anyway.
> That is your setup problem.
> >
> >
> >>>> Please allow me to provide an extreme example, is one single admin
> >>>> vq limitless, that can serve hundreds to thousands of VMs migration?
> >>> It is left to the device implementation. Just like RSS and multi queue
> support?
> >>> Is one Q enough for 800Gbps to 10Mbps link?
> >>> Answer is: Not the scope of specification, spec provide the
> >>> framework to scale
> >> this way, but not impose on the device.
> >> Even if not support RSS or MQ, the device still can work with
> >> performance overhead, not fail.
> >>
> > _work_ is subjective.
> > The financial transaction (application) failed. Packeted worked.
> > LM commands were successful, but it was not timely.
> >
> > Same same..
> >
> >> Insufficient bandwidth & resource caused live migration fail is
> >> totally different.
> > Very abstract point and unrelated to administration commands.
> It is your design facing the problem.
> >
> >>>> If not, two or
> >>>> three or what number?
> >>> It really does not matter. Its wrong point to discuss here.
> >>> Number of queues and command execution depends on the device
> >> implementation.
> >>> A financial transaction application can timeout when a device
> >>> queuing delay
> >> for virtio net rx queue is long.
> >>> And we donât put details about such things in specification.
> >>> Spec takes the requirements and provides driver device interface to
> >> implement and scale.
> >>> I still donât follow the motivation behind the question.
> >>> Is your question: How many admin queues are needed to migrate N
> >>> member
> >> devices? If so, it is implementation specific.
> >>> It is similar to how such things depend on implementation for 30
> >>> virtio device
> >> types.
> >>> And if are implying that because it is implementation specific, that
> >>> is why
> >> administration queue should not be used, but some configuration
> >> register should be used.
> >>> Than you should propose a config register interface to post
> >>> virtqueue
> >> descriptors that way for 30 device types!
> >> if so, leave it as undefined? A potential risk for device implantation?
> >
> >> Then why must the admin vq?
> > Because administration commands and admin vq does not impose devices to
> implement thousands of registers which must have time bound completion
> guarantee.
> > The large part of industry including SIOV devices led by Intel and others are
> moving away from register access mode.
> >
> > To summarize, administration commands and queue offer following benefits.
> >
> > 1. Ability to do bulk data transfer between driver and device
> >
> > 2. Ability to parallelize the work within driver and within device
> > within single or multiple virtqueues
> >
> > 3. Eliminates implementing PCI read/write MMIO registers which demand
> > low latency response interval
> >
> > 4. Better utilize host cpu as no one needs to poll on the device
> > register for completion
> >
> > 5. Ability to handle variability in command completion by device and
> > ability to notify the driver
> >
> > If this does not satisfy you, please refer to some of the past email discussions
> during administration virtuqueue time.
> I think you mixed up the facility and the implementation in my series, please
> read.
I donât know what you refer to. You asked "why AQ is must?" I answered above what AQ has to offer than some synchronous register.