RE: [virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

[Parav Pandit <parav@nvidia.com>]

> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
> open.org> On Behalf Of Michael S. Tsirkin
> Sent: Wednesday, November 8, 2023 10:46 PM
> 
> On Wed, Nov 08, 2023 at 09:00:00AM +0000, Parav Pandit wrote:
> > > From: Michael S. Tsirkin <mst@redhat.com>
> > > Sent: Wednesday, November 8, 2023 1:47 PM
> >
> > > Only for TX, and I'm pretty sure they had the foresight to test RX
> > > not just TX but let's confirm. Parav did you test both directions?
> > Rx is the main part to test for the dirty page tracking.
> > Tx only is around ring updates, so not very interesting test.
> >
> > When page tracking is started, there is impact on Rx as natural throttle by
> slowing down rx packet rate.
> 
> Wait a sec sounds kind of like what we see with SVQ as well?
> 
> > And devices have implementation choice to covering the spectrum on amount
> of drop rate.
> 
> What does this mean practically?
How to implement dirty page tracking is implementation specific detail.
So for example, what does extra things device does in its rx path to track pages, which data structures etc is used, depends on device implementation.
For example, practically, the device under test by us cannot do at 800Gbps in current generation, but possibly in future.

> 
> > And has implementations evolve, it will improve as well.
> 
> Maybe, or maybe not.
> 
> > PRI for sure is out of question. Most workload won't use it at all given other
> limitations of virtio on PCI front and existing PRI behaviors.
> 
> Out of curiousity what do you refer to?
> 
The whole scheme to notify page faults from device to iommu and serving them back is not as efficient and scalable as the way cpu can page fault.

> > There is also a test with platform IOMMU dirty page tracking done as well but
> it is very early stage to comment in this forum.
> 
> Interesting. That's a very important test I'd say - if platform based dirty page
> tracking is just as good we can avoid adding it to virtio.
I agree. I don't see all platforms have it. There is ongoing work.
> 
> 
> And I'm rather confused at this point - I was under the impression you already
> have a prototype which shows negligeable performance impact with on device
> tracking.

Compared to no tracking by platform, vs device tracking, there is significant gain.
Compared to platform tracking and device tracking, hard to say.
Given all platforms may not have it, it is not very interesting from perf point of view.