Re: [Virtio-networking] Doorbell mapping of vDPA

["Michael S. Tsirkin" <mst@redhat.com>]

On Fri, Apr 17, 2020 at 05:29:00PM +0800, Jason Wang wrote:
> 
> On 2020/4/17 äå2:37, Michael S. Tsirkin wrote:
> > On Fri, Apr 17, 2020 at 12:19:43PM +0800, Jason Wang wrote:
> > > On 2020/4/15 äå12:20, Michael S. Tsirkin wrote:
> > > > On Tue, Apr 14, 2020 at 01:12:51PM +0000, Vitaly Mireyno wrote:
> > > > > > -----Original Message-----
> > > > > > From: virtio-networking-bounces@redhat.com <virtio-networking-bounces@redhat.com> On Behalf
> > > > > > Of Jason Wang
> > > > > > Sent: Tuesday, 7 April, 2020 10:56
> > > > > > To: virtio-networking@redhat.com; Virtio-Dev <virtio-dev@lists.oasis-open.org>
> > > > > > Cc: Michael S. Tsirkin <mst@redhat.com>
> > > > > > Subject: [Virtio-networking] Doorbell mapping of vDPA
> > > > > > 
> > > > > > ----------------------------------------------------------------------
> > > > > > Hi all:
> > > > > > 
> > > > > > To get native performance of VF, we need to map doorbell to guest to avoid unnecessary vmexit. In
> > > > > > order to do this, we will launch qemu with page-per-vq=on. This means the each doorbell register
> > > > > > should be located at the beginning of 4K page and does not share the page with other registers. Then
> > > > > > vDPA framework can safely map it into the guest physical address (GPA) range defined by qemu. It
> > > > > > could be either
> > > > > > 
> > > > > > 1) a single doorbell register that is used by all virtqueues
> > > > > > 
> > > > > > or
> > > > > > 
> > > > > > 2) several different per-vq doorbell registers
> > > > > > 
> > > > > > If you decide to implement a virtio-pci register layout, need to make sure for notification structure
> > > > > > (4.1.4.4 of virtio spec):
> > > > > > 
> > > > > > For each virtqueue, the result
> > > > > > ofcap.offset+queue_notify_off*notify_off_multiplier is PAGE_SIZE (e.g
> > > > > > 4K) alignment, and the doorbeel does not share the page with other registers.
> > > > > > 
> > > > > > And it would be better if queue_notify_off, notify_off_multiplier can be changed via firmware for
> > > > > > extra flexibility.
> > > > > > 
> > > > > In some cases, these conditions could not be met for a virtio-net hardware device over PCI transport.
> > > > > queue_notify and notify_off_multiplier could not always be fully controlled by the firmware. There could be hardware limitations on flexibility degree of these parameters.
> > > > > Specifically, the limitations I'm thinking of are:
> > > > >    * queue_notify_off>0 and notify_off_multiplier>0
> > > > >    * Several doorbell registers of several virtqueues share the same page (but don't share the page with other registers).
> > > > > 
> > > > > Can this be supported in vDPA with direct doorbell mapping?
> > > > > 
> > > > > Thanks
> > > > There's value in being able to intercept some vqs in software
> > > > while the rest of vqs are handled in hardware.
> > > > E.g. that's the case for e.g. the control vq.
> > > 
> > > Good point, so in this case, the doorbell of control vq must exclusively own
> > > a page.
> > > 
> > > Or to facilitate the hardware design, we may introduce dedicated
> > > notification area for control vq?
> > > 
> > > Thanks
> > Well all this would need spec changes.
> 
> 
> Yes.
> 
> 
> > I'm guessing at this
> > point it's easier to just have hardware send commands to
> > qemu through some channel before using them.
> 
> 
> Yes, the plan is to let hardware report the location (offset) of the
> doorbell registers.
> 
> 
> >   With SRIOV that would need
> > to be either the PF, or a memory bar of the VF.
> 
> 
> So I think what we want is:
> 
> - map doorbells for non control vqs
> - intercept the control vq doorbell
> 
> In order to do this, hardware need to make sure that the doorbell of control
> vq registers does not share page with any other doorbells.
> 
> Then qemu may provide a suitable queue_notify_off for control vq which make
> sure it can be intercepted independently.
> 
> Thanks

Right. So this is something we can consider adding to the spec:
classification of VQs as data path/control, and then
a recommendation to separate at least control from data.



> 
> > 
> > 
> > > > 
> > > > > > Please check and make sure your hardware have such ability and feel free to ask if you have questions
> > > > > > (offline if necessary).
> > > > > > 
> > > > > > Thanks
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > _______________________________________________
> > > > > > Virtio-networking mailing list
> > > > > > Virtio-networking@redhat.com
> > > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__www.redhat.com_mailman_listinfo_virtio-
> > > > > > 2Dnetworking&d=DwIGaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r=lDHJ2FW52oJ3lqqsArgFRdcevq01tbLQAw
> > > > > > 4A_NO7xgI&m=pddgVCz-
> > > > > > orGTuUXATJ4Dmi7vAXatG9w47AmULNC3V9A&s=5mfYWbLCjcZO8FcwDqgAc5bjE-H-
> > > > > > 4p5TBkRZqP3uMsQ&e=