Re: [PATCH V2 6/6] virtio-pci: implement dirty page tracking

["Zhu, Lingshan" <lingshan.zhu@intel.com>]

On 11/3/2023 6:50 PM, Michael S. Tsirkin wrote:

> On Fri, Nov 03, 2023 at 06:34:37PM +0800, Zhu Lingshan wrote:
> > +\item[\field{bitmap_addr}]
> > +	The driver use this to set the address of the bitmap which records the dirty pages
> > +	caused by the device.
> > +	Each bit in the bitmap represents one memory page, bit 0 in the bitmap
> > +	reprsents page 0 at address 0, bit 1 represents page 1, and so on in a linear manner.
> > +	When \field{enable} is set to 1 and the device writes to a memory page,
> > +	the device MUST set the corresponding bit to 1 which indicating the page is dirty.
> > +\item[\field{bitmap_length}]
> > +	The driver use this to set the length in bytes of the bitmap.
> > +\end{description}
> > +
> > +\devicenormative{\subsubsection}{Memory Dirty Pages Tracker Capability}{Virtio Transport Options / Virtio Over PCI Bus / Memory Dirty Pages Tracker Capability}
> > +
> > +The device MUST NOT set any bits beyond bitmap_length when reporting dirty pages.
> > +
> > +To prevent a read-modify-write procedure, if a memory page is dirty,
> > +optionally the device is permitted to set the entire byte, which encompasses the relevant bit, to 1.
> > +
> > +The device MAY increase \field{gra_power} to reduce \field{bitmap_length}.
> > +
> > +The device must ignore any writes to \field{pasid} if PASID Extended Capability is absent or
> > +the PASID functionality is disabled in PASID Extended Capability
> >       
>
> I have to say this is going to work very badly when the number of dirty
> pages is small: you will end up scanning and re-scanning all of bitmap.
>     
    The driver needs to scan anyway, Intel
      production work with similar bitmap based dirty page tracking
      solution for years.
      
      Otherwise the device should report PFN which is not very
      practical.
    
> And the resolution is apparently 8 pages? You have just multiplied
> the migration bandwidth by a factor of 8.
>     
    No, as described in the comments, the tacking
      granularity is controlled by
\field{gra_power}, one bit represents a page with page_size = 2^(12 + gra_power).
This can also be used to reduce the size of the bitmap.
"To prevent a read-modify-write procedure, if a memory page is dirty,
optionally the device is permitted to set the entire byte, which encompasses the relevant bit, to 1."

This is optional and DMA is very likely to write a neighbor page, and the device transmit a whole byte anyway
when a bit is dirty.

How about we use platform dirty page tracking facility then implement this in virtio, as Jason suggested?
>