Re: [PATCH V2 6/6] virtio-pci: implement dirty page tracking

["Michael S. Tsirkin" <mst@redhat.com>]

On Fri, Nov 03, 2023 at 10:32:59PM +0800, Zhu, Lingshan wrote:
> 
> 
> On 11/3/2023 6:50 PM, Michael S. Tsirkin wrote:
> 
>     On Fri, Nov 03, 2023 at 06:34:37PM +0800, Zhu Lingshan wrote:
> 
>         +\item[\field{bitmap_addr}]
>         +       The driver use this to set the address of the bitmap which records the dirty pages
>         +       caused by the device.
>         +       Each bit in the bitmap represents one memory page, bit 0 in the bitmap
>         +       reprsents page 0 at address 0, bit 1 represents page 1, and so on in a linear manner.
>         +       When \field{enable} is set to 1 and the device writes to a memory page,
>         +       the device MUST set the corresponding bit to 1 which indicating the page is dirty.
>         +\item[\field{bitmap_length}]
>         +       The driver use this to set the length in bytes of the bitmap.
>         +\end{description}
>         +
>         +\devicenormative{\subsubsection}{Memory Dirty Pages Tracker Capability}{Virtio Transport Options / Virtio Over PCI Bus / Memory Dirty Pages Tracker Capability}
>         +
>         +The device MUST NOT set any bits beyond bitmap_length when reporting dirty pages.
>         +
>         +To prevent a read-modify-write procedure, if a memory page is dirty,
>         +optionally the device is permitted to set the entire byte, which encompasses the relevant bit, to 1.
>         +
>         +The device MAY increase \field{gra_power} to reduce \field{bitmap_length}.
>         +
>         +The device must ignore any writes to \field{pasid} if PASID Extended Capability is absent or
>         +the PASID functionality is disabled in PASID Extended Capability
> 
> 
>     I have to say this is going to work very badly when the number of dirty
>     pages is small: you will end up scanning and re-scanning all of bitmap.
> 
> The driver needs to scan anyway,

Not with e.g. Parav's proposal - device reports individual pages
changed. This is analogous to PML.

> Intel production work with similar bitmap
> based dirty page tracking solution for years.

and then VMs became bigger and PML was introduced.

> Otherwise the device should report PFN which is not very practical.

Why not?

>     And the resolution is apparently 8 pages? You have just multiplied
>     the migration bandwidth by a factor of 8.
> 
> No, as described in the comments, the tacking granularity is controlled by \
> field{gra_power}, one bit represents a page with page_size = 2^(12 +
> gra_power). This can also be used to reduce the size of the bitmap.

.. at the cost of increasing migration bandwidth.

> "To prevent a read-modify-write procedure, if a memory page is dirty,
> optionally the device is permitted to set the entire byte, which encompasses the relevant bit, to 1."
> 
> This is optional and DMA is very likely to write a neighbor page, and the device transmit a whole byte anyway
> when a bit is dirty.
> 
> How about we use platform dirty page tracking facility then implement this in virtio, as Jason suggested?
> 

Without something like PML it likely won't scale either.

-- 
MST