Re: [virtio-dev] Dirty Page Tracking (DPT)

[Eugenio Perez Martin <eperezma@redhat.com>]

Hi everyone.

As proposed in the previous virtio-networking Meeting, I summarized in
this document the different proposed requirements/architectures about
vDPA live migration, and a starting draft about the proposed actions.

Feedback is welcome.

Thanks!

https://docs.google.com/document/d/1-2kxRxce2CwttfsZMMoqHjyI_c-o_ZjdaHN9JHxja4M/edit?usp=sharing


On Mon, Apr 13, 2020 at 3:55 PM Jason Wang <jasowang@redhat.com> wrote:
>
>
> On 2020/4/13 äå8:15, Eugenio Perez Martin wrote:
> > On Fri, Apr 10, 2020 at 4:40 AM Jason Wang <jasowang@redhat.com> wrote:
> >>
> >> On 2020/4/10 äå5:06, Michael S. Tsirkin wrote:
> >>> On Tue, Apr 07, 2020 at 11:52:46AM +0200, Eugenio Perez Martin wrote:
> >>>> Hi!
> >>>>
> >>>> So, from the previous mails, it seems that monitoring the used ring
> >>>> (and the packed descriptors) is a good first step in that direction,
> >>>> as DPDK did. This way, the device does not need to worry about the
> >>>> dirty page tracking using a bitmap and the PCI writes limitation, and
> >>>> we can evaluate later the proposed alternatives:
> >>>> * Alternate used descriptors in packed.
> >>>> * vDPA interface for vDPA devices in a convenient format.
> >>>>
> >>>> Any thoughts? Do you think that we should start with another way?
> >>>>
> >>>> Thanks!
> >>> I am concerned that with software in data path, we'll hit RX queue
> >>> underruns, won't we?
> >>
> >> Do you mean it will lose some performance? If yes, I think so.
> >>
> >>
> >>> Two ways to avoid underruns:
> >>> - dirty page tracking
> >>> - page faults
> >>
> >> It looks to me this will lead even worse performance than software path?
> >> There will be lots of page faults during RX.
> >>
> >> Another direction is to track dirty pages via IOMMU. E.g recent Intel
> >> IOMMU has EA and D bit which could be used for tracking pages wrote by
> >> devices but not CPU.
> >>
> > So this could be added on top of the dirty tracking mechanism, isn't?
>
>
> Yes, it requires the support from IOMMU API and driver.
>
>
> > or would it be easier to start another-way around, and to start using
> > modern IOMMU and then extend to old generic code?
>
>
> If my understanding is correct, even for modern IOMMU it still require a
> lot of work. So we need to start from software support and hardware
> support first.
>
>
> >
> >>> I'm working on a proposal for page faults now.
> >>
> >> I guess it's better to have a transport independent method.
> >>
> >>
> >>> If someone wants
> >>> to work on dirty tracking in addition, that's also an option.
> >>
> >> I remember Rob mention some challenges of implementing dirty bitmap, I
> >> wonder something like queue based interface would be better (similar to
> >> Peter did for KVM)?
> >>
> > I think that the main challenge was to write bits instead of writes
> > using PCI bus.
>
>
> Yes, the ring interface will work one the descriptor which would be
> several bytes instead of bits.
>
> Thanks
>
>
> >
> >  From a conversation with Juan, another solution could be to do a byte
> > map DPT, where a byte represents a page, not a bit. While I find this
> > solution simpler, I'm not sure about the performance implications of
> > track this way (memory/TLB pressure, etc). Rob and Juan, please
> > correct me if I'm wrong or missed something.
> >
> > Compared to the used ring interposition, much more logic needs to be
> > in the device driver for both alternatives (byte mapping and queue),
> > isn't it?
> >
> >> Thanks
> >>
>