Re: [virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

["Zhu, Lingshan" <lingshan.zhu@intel.com>]

On 10/31/2023 5:41 PM, Michael S. Tsirkin wrote:
> On Tue, Oct 31, 2023 at 05:32:01PM +0800, Zhu, Lingshan wrote:
> > On 10/31/2023 3:45 PM, Michael S. Tsirkin wrote:
> > > On Tue, Oct 31, 2023 at 03:27:12AM +0000, Parav Pandit wrote:
> > > > > From: Jason Wang <jasowang@redhat.com>
> > > > > Sent: Tuesday, October 31, 2023 7:13 AM
> > > > >
> > > > > On Mon, Oct 30, 2023 at 9:21âPM Parav Pandit <parav@nvidia.com> wrote:
> > > > > > During a device migration flow (typically in a precopy phase of the
> > > > > > live migration), a device may write to the guest memory. Some
> > > > > > iommu/hypervisor may not be able to track these written pages.
> > > > > > These pages to be migrated from source to destination hypervisor.
> > > > > >
> > > > > > A device which writes to these pages, provides the page address record
> > > > > > of the to the owner device. The owner device starts write recording
> > > > > > for the device and queries all the page addresses written by the
> > > > > > device.
> > > > > >
> > > > > > Fixes: https://github.com/oasis-tcs/virtio-spec/issues/176
> > > > > > Signed-off-by: Parav Pandit <parav@nvidia.com>
> > > > > > Signed-off-by: Satananda Burla <sburla@marvell.com>
> > > > > > ---
> > > > > > changelog:
> > > > > > v1->v2:
> > > > > > - addressed comments from Michael
> > > > > > - replaced iova with physical address
> > > > > > ---
> > > > > >    admin-cmds-device-migration.tex | 15 +++++++++++++++
> > > > > >    1 file changed, 15 insertions(+)
> > > > > >
> > > > > > diff --git a/admin-cmds-device-migration.tex
> > > > > > b/admin-cmds-device-migration.tex index ed911e4..2e32f2c 100644
> > > > > > --- a/admin-cmds-device-migration.tex
> > > > > > +++ b/admin-cmds-device-migration.tex
> > > > > > @@ -95,6 +95,21 @@ \subsubsection{Device Migration}\label{sec:Basic
> > > > > > Facilities of a Virtio Device /  The owner driver can discard any
> > > > > > partially read or written device context when  any of the device migration flow
> > > > > should be aborted.
> > > > > > +During the device migration flow, a passthrough device may write data
> > > > > > +to the guest virtual machine's memory, a source hypervisor needs to
> > > > > > +keep track of these written memory to migrate such memory to destination
> > > > > hypervisor.
> > > > > > +Some systems may not be able to keep track of such memory write
> > > > > > +addresses at hypervisor level. In such a scenario, a device records
> > > > > > +and reports these written memory addresses to the owner device. The
> > > > > > +owner driver enables write recording for one or more physical address
> > > > > > +ranges per device during device migration flow. The owner driver
> > > > > > +periodically queries these written physical address records from the device.
> > > > > I wonder how PA works in this case. Device uses untranslated requests so it can
> > > > > only see IOVA. We can't mandate ATS anyhow.
> > > > Michael suggested to keep the language uniform as PA as this is ultimately what the guest driver is supplying during vq creation and in posting buffers as physical address.
> > > Yes the spec calls the address accessed by the device "physical
> > > address". Granted, this is pointless - there is only one
> > > type of address device can access. We can if we want to
> > > replace that with just "address" or "memory address".
> > > I don't think this ever caused confusion though and worth the
> > > churn.
> > But you know PA means disable IOMMU and always enable ATS on both the device
> > and host
> That is not how virtio uses the term.  Just grep "physical address" in the spec.
> This patch should be consistent.
OK, I use other terms in my series, let's see whether it is better.
>