Re: [virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

["Zhu, Lingshan" <lingshan.zhu@intel.com>]

On 11/15/2023 7:52 PM, Michael S. Tsirkin wrote:
> On Wed, Nov 15, 2023 at 04:42:56PM +0800, Zhu, Lingshan wrote:
> > On 11/15/2023 4:05 PM, Michael S. Tsirkin wrote:
> > > On Wed, Nov 15, 2023 at 03:59:16PM +0800, Zhu, Lingshan wrote:
> > > > On 11/15/2023 3:51 PM, Michael S. Tsirkin wrote:
> > > > > On Wed, Nov 15, 2023 at 12:05:59PM +0800, Zhu, Lingshan wrote:
> > > > > > On 11/14/2023 4:27 PM, Michael S. Tsirkin wrote:
> > > > > > > On Tue, Nov 14, 2023 at 03:34:32PM +0800, Zhu, Lingshan wrote:
> > > > > > > > > > So I can't
> > > > > > > > > > believe it has good performance overall. Logging via IOMMU or using
> > > > > > > > > > shadow virtqueue doesn't need any extra PCI transactions at least.
> > > > > > > > > On the other hand they have an extra CPU cost.  Personally if this is
> > > > > > > > > coming from a hardware vendor, I am inclined to trust them wrt PCI
> > > > > > > > > transactions.  But anyway, discussing this at a high level theoretically
> > > > > > > > > is pointless - whoever bothers with actual prototyping for performance
> > > > > > > > > testing wins, if no one does I'd expect a back of a napkin estimate
> > > > > > > > > to be included.
> > > > > > > > if so, Intel has released productions implementing these interfaces years
> > > > > > > > ago,
> > > > > > > > see live migration in 4.1. IFCVF vDPA Implementation,
> > > > > > > > https://doc.dpdk.org/guides-21.11/vdpadevs/ifc.html
> > > > > > > > and
> > > > > > > That one is based on shadow queue, right? Which I think this shows
> > > > > > > is worth supporting.
> > > > > > Yes, it is shadow virtqueue, I assume this is already mostly done,
> > > > > > do you see any gaps we need to address in our series that we should work on?
> > > > > >
> > > > > > Thanks
> > > > > There were a ton of comments posted on your series.
> > > > Hope I didn't miss anything. I see your latest comments are about vq states,
> > > > as replied before, I think we can record the states by two le16 and the
> > > > in-flight
> > > > descriptor tracking facility.
> > > I don't know why you need the le16. in-flight tracking should be enough.
> > > And given it needs DMA I would try really hard to actually use
> > > admin commands for this.
> > we need to record the on-device avail_idx and used_idx, or
> > how can the destination side know the device internal values.
> Again you never documented what state the device is in so I can't really
> say for sure.  But generally whenever a buffer is used the internal
> values are written out to memory.
This is the state of a virtqueue, in my series I have defined what is
vq state in [PATCH V2 1/6] virtio: introduce virtqueue state,
and give an example of PCI implementation.

In short, for split vq it is last_avail_idx and in-flight descriptors.

I humbly request an explicit list of missing gaps, so that I can improve 
my V3

Thanks
> > > > For this shadow virtqueue, do you think I should address this in my V4?
> > > > Like saying: acknowledged control commands through the control virtqueue
> > > > should be recorded, and we want to use shadow virtqueue to track dirty
> > > > pages.
> > > What you need to do is actually describe what do you expect the device
> > > to do when it enters this suspend state. since you mention control
> > > virtqueue then it seems that there needs to be a device type
> > > specific text explaining the behaviour. If so this implies there
> > > needs to be a list of device types that support suspend
> > > until someone looks at each type and documents what it does.
> > On a second thought, shadow vqs are hypervisor behaviors, maybe should not
> > be
> > described in this device spec.
> >
> > Since SUSPEND is in device status, so for now I see every type of device
> > implements
> > device_status should support SUSPEND. This should be a general facility.