Re: [virtio-comment] Re: [PATCH 0/2] Selective queue enabling

["Michael S. Tsirkin" <mst@redhat.com>]

On Thu, Jun 08, 2023 at 08:44:41AM +0800, Jason Wang wrote:
> On Thu, Jun 8, 2023 at 4:27âAM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Wed, Jun 07, 2023 at 11:41:39AM +0200, Eugenio Perez Martin wrote:
> > > On Wed, Jun 7, 2023 at 10:59âAM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > >
> > > > On Wed, Jun 07, 2023 at 10:47:12AM +0200, Eugenio Perez Martin wrote:
> > > > > On Wed, Jun 7, 2023 at 10:23âAM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > > > > >
> > > > > > On Wed, 7 Jun 2023 07:35:58 +0200, Eugenio Perez Martin <eperezma@redhat.com> wrote:
> > > > > > > On Tue, Jun 6, 2023 at 9:10âPM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > > >
> > > > > > > > On Tue, Jun 06, 2023 at 07:55:09PM +0200, Eugenio PÃrez wrote:
> > > > > > > > > This series allows the driver to start the device (as set DRIVER_OK) with only
> > > > > > > > > some queues enabled, and then enable another queues later.
> > > > > > > > >
> > > > > > > > > This is the current way to migrate net device state through control
> > > > > > > > > virtqueue, in a software assisted framework with vDPA:
> > > > > > > > > * First, only net CVQ is enabled at DRIVER_OK
> > > > > > > > > * All the control commands (mac address, mq, etc) needed for the device
> > > > > > > > > to behave the same as the source of migration are sent
> > > > > > > > > * Finally all the dataplane queues are enabled.
> > > > > > > >
> > > > > > > > In my opinion, this is somewhat problematic. Specifically, currently
> > > > > > > > devices tend to deduce how many queues are needed by looking
> > > > > > > > at the state at DRIVER_OK time.
> > > > > > > >
> > > > > > > > Question: what is wrong with enabling queues initially and then
> > > > > > > > doing a reset right after DRIVER_OK? You can even allocate
> > > > > > > > memory for just one queue (zeroing it out).
> > > > > > > >
> > > > > > > > Granted this looks kind of ugly but side-steps this problem with
> > > > > > > > no need for spec changes.
> > > > > > > >
> > > > > > >
> > > > > > > The problem is that the rx queues can start receiving, as the guest
> > > > > > > already has buffers there.
> > > > > >
> > > > > > Can we reset the vq before filling buffers to it?
> > > > > >
> > > > >
> > > > > They are passthrough from the guest so there is a window where the
> > > > > device can process rx descriptors.
> > > >
> > > > Not if there are no descriptors there.
> > > >
> > >
> > > But the migration is driven by the hypervisor, and it cannot control
> > > that. The guest will likely have rx descriptors available.
> >
> > Maybe I misunderstand. Is hypervisor driving cvq while guest is driving
> > rx queues?
> 
> No, hypervisor tries to restore virtqueue states via cvq before guest can drive.

So cvq maps to hypervisor memory?


> So in this case if RX queues are enabled at the same time, the device
> may try to queue packets to queue 0.
> 
> Thanks
> 
> > How do you do this - they are DMA from same VF no?
> >
> >
> > > >
> > > > > > > Apart from that, the back and forth
> > > > > > > introduces latencies.
> > > > > > >
> > > > > > > Maybe a better angle is to start all the queues as if they're reset,
> > > > > > > write 1 just to CVQ, configure the device, and then write 1 to all
> > > > > > > dataplane vqs?
> > > > > >
> > > > > > write to what?
> > > > > >
> > > > >
> > > > > Sorry I was unclear, I mean to enable the vqs writing 1 to queue_enable.
> > > > >
> > > > > Thanks!
> > > > >
> > > > > > Thanks.
> > > > > >
> > > > > > >
> > > > > > > Thanks!
> > > > > > >
> > > > > > > > > Eugenio PÃrez (2):
> > > > > > > > >   virtio: introduce selective queue enabling
> > > > > > > > >   virtio: pci support virtqueue selective enabling
> > > > > > > > >
> > > > > > > > >  content.tex       | 15 +++++++++++++--
> > > > > > > > >  transport-pci.tex |  4 ++++
> > > > > > > > >  2 files changed, 17 insertions(+), 2 deletions(-)
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > 2.31.1
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > > This publicly archived list offers a means to provide input to the
> > > > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > > > >
> > > > > > In order to verify user consent to the Feedback License terms and
> > > > > > to minimize spam in the list archive, subscription is required
> > > > > > before posting.
> > > > > >
> > > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > > > Join OASIS: https://www.oasis-open.org/join/
> > > > > >
> > > >
> >