Re: [virtio-dev] Re: [PATCH 0/5] virtio: introduce SUSPEND bit and vq state

[Jason Wang <jasowang@redhat.com>]

On Thu, Sep 21, 2023 at 12:19âPM Parav Pandit <parav@nvidia.com> wrote:
>
>
> > From: Jason Wang <jasowang@redhat.com>
> > Sent: Thursday, September 21, 2023 9:39 AM
> >
> > On Thu, Sep 21, 2023 at 12:01âPM Parav Pandit <parav@nvidia.com> wrote:
> > >
> > >
> > >
> > > > From: Jason Wang <jasowang@redhat.com>
> > > > Sent: Thursday, September 21, 2023 8:48 AM
> > >
> > > > As replied in another thread, the issues for BAR are:
> > > >
> > > > 1) Not sure it can have an efficient interface, it would be
> > > > something like VIRTIO_PCI_CAP_PCI_CFG which is very slow compared to
> > > > single register accessing
> > > > 2) There's no owner/group/member for MMIO, most of the time, we only
> > > > need a single MMIO device. If we want the owner to manage itself, it
> > > > seems redundant as is implied in all the existing transports (without admin
> > commands).
> > > > Even if we had, it might still suffer from bootstrap issues.
> > > > 3) For live migration, it means the admin commands needs to start
> > > > from duplicating every existing transport specific interface it can
> > > > give us. One example is that we may end up with two interfaces to
> > > > access virtqueue addresses etc. This results in extra complicity and
> > > > it is actually a full transport (driver can just use admin commands to drive
> > the device).
> > > In [1] there is no duplication. The live migration driver never parses the device
> > context either while reading or write.
> > > Hence no code and no complexity in driver and no duplicate work.
> > > Therefore, those admin commands are not to drive the guest device either.
> >
> > I'm not sure how this is related to the duplication issue.
> >
> You commented that admin virtqueue duplicates somethings.
> And I explained above that it does not.
>
> > >
> > > > 4) Admin commands itself may not be capable of doing things like
> > > > dirty page logging, it requires the assistance from the transport
> > > >
> > > Admin command in [1] is capable of dirty page logging.
> >
> > In your design, the logging is done via DMA not the virtqueue.
> >
> No. it is done via admin command, not DMA in [2].
>
> [2] https://lore.kernel.org/virtio-comment/20230909142911.524407-7-parav@nvidia.com/T/#m17b09acd8c73d374e98ad84764b315afa94f59c9
>
> > The only job for virtqueue is to initiate the DMA. But if DMA can be initiated via
> > virtqueue, it can be done in other ways.
> >
> Lets first establish 4 things in alternative way, any motivation to do so with 5th point without giant registers need in device.
>
> > >
> > > > 1) Parav's proposal does several couplings: couple basic build
> > > > blocks (suspend, dirty page tracking) with live migration, couple
> > > > live migration with admin commands.
> > > In which use case you find dirty page tracking useful without migration for
> > which you like to see it detached from device migration flow?
> >
> > Is it only the dirty page tracking? It's the combinations of
> >
> > 1) suspending
> > 2) device states
> > 3) dirty page tracking
> >
> > Eeah of those will have use cases other than live migration: VM stop, power
> > management in VM, profiling and monitoring, failover etc.
> >
> Suspend/resume with different power state is driven by the guest directly.

And there's hibernation actually where device states might be useful.

> So it may find some overlap.
>
> Device context has no overlap.

I can give you one example, e.g debugging.

>
> Dirty page tracking has no overlap. What do you want to profile and monitor? In case if you want to profile, it can be used without migration command anyway?

It works like a dirty bit of PTE. We all know it has a broader use
case than logging. For example, tracking working set and do
optimization on IOMMU/IOTLB or even device IOTLB.

1) Try to prove your facility can only work for one specific cases
2) Try to prove your facility can work for more than one cases

Which one is easier and more beneficial to virtio?


> If you describe, may be we I can split "device migration" chapter to two pieces,
> Device management and device migration.
>
> Device migration will use these basic facility.
> Would that help you?

Definitely, but it needs to be done by not making it under the
subsection of admin commands, that's it.

Let me repeat once again here for the possible steps to collaboration:

1) define virtqueue state, inflight descriptors in the section of
basic facility but not under the admin commands
2) define the dirty page tracking, device context/states in the
section of basic facility but not under the admin commands
3) define transport specific interfaces or admin commands to access them

Does this work? It seems you refused such steps in the past.

Actually, I would like to leave 2) as it's very complicated which
might not converge easily.

Thanks