Re: [virtio-comment] Re: [PATCH V2 6/6] virtio-pci: implement dirty page tracking

["Zhu, Lingshan" <lingshan.zhu@intel.com>]

On 11/6/2023 12:16 AM, Michael S. Tsirkin wrote:
> On Fri, Nov 03, 2023 at 10:32:59PM +0800, Zhu, Lingshan wrote:
> > On 11/3/2023 6:50 PM, Michael S. Tsirkin wrote:
> >
> >      On Fri, Nov 03, 2023 at 06:34:37PM +0800, Zhu Lingshan wrote:
> >
> >          +\item[\field{bitmap_addr}]
> >          +       The driver use this to set the address of the bitmap which records the dirty pages
> >          +       caused by the device.
> >          +       Each bit in the bitmap represents one memory page, bit 0 in the bitmap
> >          +       reprsents page 0 at address 0, bit 1 represents page 1, and so on in a linear manner.
> >          +       When \field{enable} is set to 1 and the device writes to a memory page,
> >          +       the device MUST set the corresponding bit to 1 which indicating the page is dirty.
> >          +\item[\field{bitmap_length}]
> >          +       The driver use this to set the length in bytes of the bitmap.
> >          +\end{description}
> >          +
> >          +\devicenormative{\subsubsection}{Memory Dirty Pages Tracker Capability}{Virtio Transport Options / Virtio Over PCI Bus / Memory Dirty Pages Tracker Capability}
> >          +
> >          +The device MUST NOT set any bits beyond bitmap_length when reporting dirty pages.
> >          +
> >          +To prevent a read-modify-write procedure, if a memory page is dirty,
> >          +optionally the device is permitted to set the entire byte, which encompasses the relevant bit, to 1.
> >          +
> >          +The device MAY increase \field{gra_power} to reduce \field{bitmap_length}.
> >          +
> >          +The device must ignore any writes to \field{pasid} if PASID Extended Capability is absent or
> >          +the PASID functionality is disabled in PASID Extended Capability
> >
> >
> >      I have to say this is going to work very badly when the number of dirty
> >      pages is small: you will end up scanning and re-scanning all of bitmap.
> >
> > The driver needs to scan anyway,
> Not with e.g. Parav's proposal - device reports individual pages
> changed. This is analogous to PML.
In this proposal, the device DMA writes the bitmap to recording dirty pages,
this can easily be merged into QEMU migration flow.
> > Intel production work with similar bitmap
> > based dirty page tracking solution for years.
> and then VMs became bigger and PML was introduced.
So you agree we should track dirty pages through the platform facilities?
I am glad to hear that!
> > Otherwise the device should report PFN which is not very practical.
> Why not?
Really? the device report PFN?
What can happen if the device keep writing a small piece of memory???
> >      And the resolution is apparently 8 pages? You have just multiplied
> >      the migration bandwidth by a factor of 8.
> >
> > No, as described in the comments, the tacking granularity is controlled by \
> > field{gra_power}, one bit represents a page with page_size = 2^(12 +
> > gra_power). This can also be used to reduce the size of the bitmap.
> .. at the cost of increasing migration bandwidth.
The device is very likely to write a neighbor page, and this happens
everywhere for example CPU read 64 bytes aligned data.

This is a tradeoff
> > "To prevent a read-modify-write procedure, if a memory page is dirty,
> > optionally the device is permitted to set the entire byte, which encompasses the relevant bit, to 1."
> >
> > This is optional and DMA is very likely to write a neighbor page, and the device transmit a whole byte anyway
> > when a bit is dirty.
> >
> > How about we use platform dirty page tracking facility then implement this in virtio, as Jason suggested?
> Without something like PML it likely won't scale either.
So that would be platform issue which we don't need to take care of?
Intel VT-d can do this for sure.
>