Re: [virtio-dev] Dirty Page Tracking (DPT)

[Jason Wang <jasowang@redhat.com>]

On 2020/4/13 äå9:30, Rob Miller wrote:
> On Mon, Apr 13, 2020 at 8:16 AM Eugenio Perez Martin 
> <eperezma@redhat.com <mailto:eperezma@redhat.com>> wrote:
>
>     On Fri, Apr 10, 2020 at 4:40 AM Jason Wang <jasowang@redhat.com
>     <mailto:jasowang@redhat.com>> wrote:
>     >
>     >
>     > On 2020/4/10 äå5:06, Michael S. Tsirkin wrote:
>     > > On Tue, Apr 07, 2020 at 11:52:46AM +0200, Eugenio Perez Martin
>     wrote:
>     > >> Hi!
>     > >>
>     > >> So, from the previous mails, it seems that monitoring the
>     used ring
>     > >> (and the packed descriptors) is a good first step in that
>     direction,
>     > >> as DPDK did. This way, the device does not need to worry
>     about the
>     > >> dirty page tracking using a bitmap and the PCI writes
>     limitation, and
>     > >> we can evaluate later the proposed alternatives:
>     > >> * Alternate used descriptors in packed.
>     > >> * vDPA interface for vDPA devices in a convenient format.
>     > >>
>     > >> Any thoughts? Do you think that we should start with another way?
>     > >>
>     > >> Thanks!
>     > > I am concerned that with software in data path, we'll hit RX queue
>     > > underruns, won't we?
>     >
>     >
>     > Do you mean it will lose some performance? If yes, I think so.
>     >
>     >
>     > > Two ways to avoid underruns:
>     > > - dirty page tracking
>     > > - page faults
>     >
>     >
>     > It looks to me this will lead even worse performance than
>     software path?
>     > There will be lots of page faults during RX.
>     >
>     > Another direction is to track dirty pages via IOMMU. E.g recent
>     Intel
>     > IOMMU has EA and D bit which could be used for tracking pages
>     wrote by
>     > devices but not CPU.
>     >
>
>     So this could be added on top of the dirty tracking mechanism, isn't?
>     or would it be easier to start another-way around, and to start using
>     modern IOMMU and then extend to old generic code?
>
>     >
>     > >
>     > > I'm working on a proposal for page faults now.
>     >
>     >
>     > I guess it's better to have a transport independent method.
>     >
>     >
>     > > If someone wants
>     > > to work on dirty tracking in addition, that's also an option.
>     >
>     >
>     > I remember Rob mention some challenges of implementing dirty
>     bitmap, I
>     > wonder something like queue based interface would be better
>     (similar to
>     > Peter did for KVM)?
>     >
>
>     I think that the main challenge was to write bits instead of writes
>     using PCI bus.
>
>     From a conversation with Juan, another solution could be to do a byte
>     map DPT, where a byte represents a page, not a bit. While I find this
>     solution simpler, I'm not sure about the performance implications of
>     track this way (memory/TLB pressure, etc). Rob and Juan, please
>     correct me if I'm wrong or missed something.

It would lead large working set, but the actual impact may require 
benchmark.


> RJM>] You captured it correctly. We (Broadcom) discussed the idea of
> using a byte (being 0xFF) which effectively indicate that 64KB were dirty,
> this would cause potentially refreshing pages to the remote that weren't
> needed to be refreshed, slowing down the migration. I like the idea of 
> mapping
> a byte to a page, and teach QEMU (or vDPA, or ...) how to interpret.


That's a way to go, I think we can try to propose an API and move the 
discussion upstream.


>     Compared to the used ring interposition, much more logic needs to be
>     in the device driver for both alternatives (byte mapping and queue),
>     isn't it?
>
> RJM> Yes, but this would (should) lead to best performance.


I agree, I think we can start from software support and discuss the 
software path in parallel.

Thanks


>     > Thanks
>     >
>
>
>     ---------------------------------------------------------------------
>     To unsubscribe, e-mail:
>     virtio-dev-unsubscribe@lists.oasis-open.org
>     <mailto:virtio-dev-unsubscribe@lists.oasis-open.org>
>     For additional commands, e-mail:
>     virtio-dev-help@lists.oasis-open.org
>     <mailto:virtio-dev-help@lists.oasis-open.org>