Re: [virtio-comment] Re: [PATCH 0/2] Selective queue enabling

[Jason Wang <jasowang@redhat.com>]

On Thu, Jun 8, 2023 at 4:27âAM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Wed, Jun 07, 2023 at 11:41:39AM +0200, Eugenio Perez Martin wrote:
> > On Wed, Jun 7, 2023 at 10:59âAM Michael S. Tsirkin <mst@redhat.com> wrote:
> > >
> > > On Wed, Jun 07, 2023 at 10:47:12AM +0200, Eugenio Perez Martin wrote:
> > > > On Wed, Jun 7, 2023 at 10:23âAM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > > > >
> > > > > On Wed, 7 Jun 2023 07:35:58 +0200, Eugenio Perez Martin <eperezma@redhat.com> wrote:
> > > > > > On Tue, Jun 6, 2023 at 9:10âPM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > >
> > > > > > > On Tue, Jun 06, 2023 at 07:55:09PM +0200, Eugenio PÃrez wrote:
> > > > > > > > This series allows the driver to start the device (as set DRIVER_OK) with only
> > > > > > > > some queues enabled, and then enable another queues later.
> > > > > > > >
> > > > > > > > This is the current way to migrate net device state through control
> > > > > > > > virtqueue, in a software assisted framework with vDPA:
> > > > > > > > * First, only net CVQ is enabled at DRIVER_OK
> > > > > > > > * All the control commands (mac address, mq, etc) needed for the device
> > > > > > > > to behave the same as the source of migration are sent
> > > > > > > > * Finally all the dataplane queues are enabled.
> > > > > > >
> > > > > > > In my opinion, this is somewhat problematic. Specifically, currently
> > > > > > > devices tend to deduce how many queues are needed by looking
> > > > > > > at the state at DRIVER_OK time.
> > > > > > >
> > > > > > > Question: what is wrong with enabling queues initially and then
> > > > > > > doing a reset right after DRIVER_OK? You can even allocate
> > > > > > > memory for just one queue (zeroing it out).
> > > > > > >
> > > > > > > Granted this looks kind of ugly but side-steps this problem with
> > > > > > > no need for spec changes.
> > > > > > >
> > > > > >
> > > > > > The problem is that the rx queues can start receiving, as the guest
> > > > > > already has buffers there.
> > > > >
> > > > > Can we reset the vq before filling buffers to it?
> > > > >
> > > >
> > > > They are passthrough from the guest so there is a window where the
> > > > device can process rx descriptors.
> > >
> > > Not if there are no descriptors there.
> > >
> >
> > But the migration is driven by the hypervisor, and it cannot control
> > that. The guest will likely have rx descriptors available.
>
> Maybe I misunderstand. Is hypervisor driving cvq while guest is driving
> rx queues?

No, hypervisor tries to restore virtqueue states via cvq before guest can drive.

So in this case if RX queues are enabled at the same time, the device
may try to queue packets to queue 0.

Thanks

> How do you do this - they are DMA from same VF no?
>
>
> > >
> > > > > > Apart from that, the back and forth
> > > > > > introduces latencies.
> > > > > >
> > > > > > Maybe a better angle is to start all the queues as if they're reset,
> > > > > > write 1 just to CVQ, configure the device, and then write 1 to all
> > > > > > dataplane vqs?
> > > > >
> > > > > write to what?
> > > > >
> > > >
> > > > Sorry I was unclear, I mean to enable the vqs writing 1 to queue_enable.
> > > >
> > > > Thanks!
> > > >
> > > > > Thanks.
> > > > >
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > > > Eugenio PÃrez (2):
> > > > > > > >   virtio: introduce selective queue enabling
> > > > > > > >   virtio: pci support virtqueue selective enabling
> > > > > > > >
> > > > > > > >  content.tex       | 15 +++++++++++++--
> > > > > > > >  transport-pci.tex |  4 ++++
> > > > > > > >  2 files changed, 17 insertions(+), 2 deletions(-)
> > > > > > > >
> > > > > > > > --
> > > > > > > > 2.31.1
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > > This publicly archived list offers a means to provide input to the
> > > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > > >
> > > > > In order to verify user consent to the Feedback License terms and
> > > > > to minimize spam in the list archive, subscription is required
> > > > > before posting.
> > > > >
> > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > > Join OASIS: https://www.oasis-open.org/join/
> > > > >
> > >
>