Re: [virtio-dev] Dirty Page Tracking (DPT)

[Jason Wang <jasowang@redhat.com>]

On 2020/3/10 äå2:24, Michael S. Tsirkin wrote:
> On Tue, Mar 10, 2020 at 11:22:00AM +0800, Jason Wang wrote:
> > On 2020/3/9 äå6:13, Michael S. Tsirkin wrote:
> > > On Mon, Mar 09, 2020 at 04:50:43PM +0800, Jason Wang wrote:
> > > > On 2020/3/9 äå3:38, Michael S. Tsirkin wrote:
> > > > > On Fri, Mar 06, 2020 at 10:40:13AM -0500, Rob Miller wrote:
> > > > > > I understand that DPT isn't really on the forefront of the vDPA framework, but
> > > > > > wanted to understand if there any initial thoughts on how this would work...
> > > > > And judging by the next few chapters, you are actually
> > > > > talking about vhost pci, right?
> > > > >
> > > > > > In the migration framework, in its simplest form, (I gather) its QEMU via KVM
> > > > > > that is reading the dirty page table, converting bits to page numbers, then
> > > > > > flushing remote VM/copying local page(s)->remote VM, ect.
> > > > > >
> > > > > > While this is fine for a VM (say VM1) dirtying its own memory and the accesses
> > > > > > are trapped in the kernel as well as the log is being updated, I'm not sure
> > > > > > what happens in the situationÂof vhost, where a remote VM (say VM2) is dirtying
> > > > > > up VM1's memory since it can directly access it, during packet reception for
> > > > > > example.
> > > > > > Whatever technique is employedÂto catch this, how would this differ from a HW
> > > > > > based Virtio device doing DMA directly into a VM's DDR, wrt to DPT? Is QEMU
> > > > > > going to have a 2nd place to query the dirty logs - ie: the vDPA layer?
> > > > > I don't think anyone has a good handle at the vhost pci migration yet.
> > > > > But I think a reasonable way to handle that would be to
> > > > > activate dirty tracking in VM2's QEMU.
> > > > >
> > > > > And then VM2's QEMU would periodically copy the bits to the log - does
> > > > > this sound right?
> > > > >
> > > > > > Further I heard about a SW based DPT within the vDPA framework for those
> > > > > > devices that do not (yet) support DPT inherently in HW. How is this envisioned
> > > > > > to work?
> > > > > What I am aware of is simply switching to a software virtio
> > > > > for the duration of migration. The software can be pretty simple
> > > > > since the formats match: just copy available entries to device ring,
> > > > > and for used entries, see a used ring entry, mark page
> > > > > dirty and then copy used entry to guest ring.
> > > > That looks more heavyweight than e.g just relay used ring (as what dpdk did)
> > > > I believe?
> > > That works for used but not for the packed ring.
> > For packed ring, we can relay the descriptor ring?
> Yes, and thus one must relay both available and used descriptors.

Yes.


> It's an interesting tradeoff. Packed ring at least was not designed
> with multiple actors in mind.


Yes.


> If this becomes a thing (and that's a big if) it might make sense to
> support temporarily reporting used entries in a separate buffer, while
> migration is in progress.  Also if doing this, it looks like we can then
> support used ring resize too, and thus it might also make sense to use
> this to support sharing a used ring between multiple available rings -
> this way a single CPU can handle multiple used rings efficiently.


Right, that's something similar to the two ring model I proposed in the 
past.

Thanks