Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

["Michael S. Tsirkin" <mst@redhat.com>]

On Thu, Nov 16, 2023 at 05:29:49PM +0000, Parav Pandit wrote:
> 
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Thursday, November 16, 2023 10:56 PM
> > 
> > On Thu, Nov 16, 2023 at 04:26:53PM +0000, Parav Pandit wrote:
> > > > From: Michael S. Tsirkin <mst@redhat.com>
> > > > Sent: Thursday, November 16, 2023 5:18 PM
> > > >
> > > > On Thu, Nov 16, 2023 at 07:40:57AM +0000, Parav Pandit wrote:
> > > > >
> > > > > > From: Michael S. Tsirkin <mst@redhat.com>
> > > > > > Sent: Thursday, November 16, 2023 1:06 PM
> > > > > >
> > > > > > On Thu, Nov 16, 2023 at 12:51:40AM -0500, Michael S. Tsirkin wrote:
> > > > > > > On Thu, Nov 16, 2023 at 05:29:54AM +0000, Parav Pandit wrote:
> > > > > > > > We should expose a limit of the device in the proposed
> > > > > > WRITE_RECORD_CAP_QUERY command, that how much range it can
> > track.
> > > > > > > > So that future provisioning framework can use it.
> > > > > > > >
> > > > > > > > I will cover this in v5 early next week.
> > > > > > >
> > > > > > > I do worry about how this can even work though. If you want a
> > > > > > > generic device you do not get to dictate how much memory VM has.
> > > > > > >
> > > > > > > Aren't we talking bit per page? With 1TByte of memory to track
> > > > > > > -> 256Gbit -> 32Gbit -> 8Gbyte per VF?
> > > > > >
> > > > > > Ugh. Actually of course:
> > > > > > With 1TByte of memory to track -> 256Mbit -> 32Mbit -> 8Mbyte
> > > > > > per VF
> > > > > >
> > > > > > 8Gbyte per *PF* with 1K VFs.
> > > > > >
> > > > > Device may not maintain as a bitmap.
> > > >
> > > > However you maintain it, there's 256Mega bit of information.
> > > There may be other data structures that device may deploy as for example
> > hash or tree or something else.
> > 
> > Point being?
> The device may have some hashing accelerator or other improvements that may perform better than bitmap as many queues in parallel attempt to update the shared database.

Maybe, I didn't give this thought.

My point was that to be able to keep all combinations of dirty/non dirty
page for each 4k page in a 1TByte guest device needs 8MBytes of
on-device memory per VF. As designed the query also has to report it for
each VF accurately even if multiple VFs are accessing same guest.

> > 
> > > And this is runtime memory only during the short live migration period of
> > 400msec or less.
> > > It is not some _always_ resident memory.
> > 
> > No - write tracking is used in the live phase of migration. It can be enabled as
> > long as you wish - it's a question of policy.  There actually exist solutions that
> > utilize this phase for redundancy, permanently running in this mode.
> 
> If such use case exists, one may further improve the device implementation.

Yes such use cases exist, there is no limit on how long migration takes.
So go ahead and further improve it please. Do not give us "we did not
get requests for this feature" please.

-- 
MST