Re: [virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

["Michael S. Tsirkin" <mst@redhat.com>]

On Thu, Nov 23, 2023 at 05:19:06AM -0800, Si-Wei Liu wrote:
> If anything relevant I would more like to see performance comparison with
> platform dirty tracking via IOMMUFD, but that's perhaps too early stage at
> this point to conclude anything given there's very limited availability (in
> terms of supporting software, I know some supporting hardware has been
> around for a few years) and none of the potential software optimizations is
> in place at this point to make a fair comparison for.


Exactly. I suggested shadow as a kind of fallback since it is instead
available.

> Granted device
> assisted tracking has its own set of limitations e.g. loose coupling or
> integration with platform features, lack of nested and PASID support et al.
> However, state of the art for platform dirty tracking is not perfect either,
> far off being highly optimized for all types of workload or scenarios. At
> least to me the cost of page table walk to scan all PTEs across all levels
> is not easily negligible - given no PML equivalent here, are we sure the
> whole range scan can be as efficient and scalable as memory size / # of PTEs
> grows? How large it may impact the downtime with this rudimentary dirty
> scan? No data point was given thus far. If chances are that there could be
> major improvement from device tracking for those general use cases to
> supplement what platform cannot achieve efficiently enough, it's not too
> good to kill off the possibility entirely at this early stage. Maybe a PoC
> or some comparative performance data can help prove the theory?
> 
> On the other hand, the device assisted tracking has at least one advantage
> that platform cannot simply offer - throttle down device for convergence,
> inherently or explicitly whenever needed. I think earlier Micheal suggested
> something to make the core data structure used for logging more efficient
> and compact, working like PML but using a queue or an array, and the entry
> of which may contain a list of discrete pages or contiguous PFN ranges.

Hmm no, what I really meant is a way for device to store all or parts
of this structure in host RAM as opposed to all of it in on-device
memory as it has to be now.

> On
> top of this one may add parallelism to distribute load to multiple queues,
> or add zero copy to speed up dirty sync to userspace - things virtio queues
> are pretty good at doing. After all, nothing can be perfect to begin with,
> and every complex feature would need substantial time to improve and evolve.
> It does so for shadow virtqueue from where it gets started to where it is
> now, even so there's still a lot of optimization work not done yet. There
> must be head room here for device page tracking or platform tracking, too.
> 
> Regards,
> -Siwei
> 
> 
> > 
> > Shadow virtqueue can be used with a save/load model for device state
> > recovery for sure.
> > 
> > > > But it seems the shadow virtqueue itself is not the major factor but
> > > > the time spent on programming vendor specific mappings for example.
> > > Yep. The slowness on mapping part is mostly due to the artifact of
> > > software-based implementation. IMHO for live migration p.o.v it's better
> > > to not involve any mapping operation in the down time path at all.
> > Yes.
> > 
> > Thanks
> > 
> > > -Siwei
> > > > Thanks
> > > > 
> > > > > The time didn't come from pci side or boot side.
> > > > > 
> > > > > For pci side of things you would want to compare the pci vs non pci device based VM boot time.
> > > > > 
> > > > This publicly archived list offers a means to provide input to the
> > > > 
> > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > > 
> > > > 
> > > > 
> > > > In order to verify user consent to the Feedback License terms and
> > > > 
> > > > to minimize spam in the list archive, subscription is required
> > > > 
> > > > before posting.
> > > > 
> > > > 
> > > > 
> > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > 
> > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > 
> > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > 
> > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > 
> > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > 
> > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > 
> > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > 
> > > > Join OASIS: https://www.oasis-open.org/join/
> > > > 
> > > 
> > > This publicly archived list offers a means to provide input to the
> > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > 
> > > In order to verify user consent to the Feedback License terms and
> > > to minimize spam in the list archive, subscription is required
> > > before posting.
> > > 
> > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > List help: virtio-comment-help@lists.oasis-open.org
> > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > Committee: https://www.oasis-open.org/committees/virtio/
> > > Join OASIS: https://www.oasis-open.org/join/
> > > 
> > 
> > This publicly archived list offers a means to provide input to the
> > 
> > OASIS Virtual I/O Device (VIRTIO) TC.
> > 
> > 
> > 
> > In order to verify user consent to the Feedback License terms and
> > 
> > to minimize spam in the list archive, subscription is required
> > 
> > before posting.
> > 
> > 
> > 
> > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > 
> > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > 
> > List help: virtio-comment-help@lists.oasis-open.org
> > 
> > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > 
> > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > 
> > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > 
> > Committee: https://www.oasis-open.org/committees/virtio/
> > 
> > Join OASIS: https://www.oasis-open.org/join/
> >