[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]
Subject: Re: [virtio-dev] Re: [Virtio-networking] Doorbell mapping of vDPA
On 2020/4/17 äå7:00, Michael S. Tsirkin wrote:
On Fri, Apr 17, 2020 at 06:25:30PM +0800, Jason Wang wrote:On 2020/4/17 äå6:06, Michael S. Tsirkin wrote:On Fri, Apr 17, 2020 at 05:59:29PM +0800, Jason Wang wrote:On 2020/4/17 äå5:37, Michael S. Tsirkin wrote:On Fri, Apr 17, 2020 at 05:31:20PM +0800, Jason Wang wrote:On 2020/4/17 äå2:39, Michael S. Tsirkin wrote:On Fri, Apr 17, 2020 at 12:22:04PM +0800, Jason Wang wrote:On 2020/4/17 äå12:19, Jason Wang wrote:On 2020/4/15 äå12:20, Michael S. Tsirkin wrote:On Tue, Apr 14, 2020 at 01:12:51PM +0000, Vitaly Mireyno wrote:-----Original Message----- From:firstname.lastname@example.org <email@example.com> On Behalf Of Jason Wang Sent: Tuesday, 7 April, 2020 10:56 To:firstname.lastname@example.org; Virtio-Dev <email@example.com> Cc: Michael S. Tsirkin<firstname.lastname@example.org> Subject: [Virtio-networking] Doorbell mapping of vDPA ---------------------------------------------------------------------- Hi all: To get native performance of VF, we need to map doorbell to guest to avoid unnecessary vmexit. In order to do this, we will launch qemu with page-per-vq=on. This means the each doorbell register should be located at the beginning of 4K page and does not share the page with other registers. Then vDPA framework can safely map it into the guest physical address (GPA) range defined by qemu. It could be either 1) a single doorbell register that is used by all virtqueues or 2) several different per-vq doorbell registers If you decide to implement a virtio-pci register layout, need to make sure for notification structure (220.127.116.11 of virtio spec): For each virtqueue, the result ofcap.offset+queue_notify_off*notify_off_multiplier is PAGE_SIZE (e.g 4K) alignment, and the doorbeel does not share the page with other registers. And it would be better if queue_notify_off, notify_off_multiplier can be changed via firmware for extra flexibility.In some cases, these conditions could not be met for a virtio-net hardware device over PCI transport. queue_notify and notify_off_multiplier could not always be fully controlled by the firmware. There could be hardware limitations on flexibility degree of these parameters. Specifically, the limitations I'm thinking of are: Â * queue_notify_off>0 and notify_off_multiplier>0 Â * Several doorbell registers of several virtqueues share the same page (but don't share the page with other registers). Can this be supported in vDPA with direct doorbell mapping? ThanksThere's value in being able to intercept some vqs in software while the rest of vqs are handled in hardware. E.g. that's the case for e.g. the control vq.Good point, so in this case, the doorbell of control vq must exclusively own a page.Or we need intercept the doorbells that share a page with control vq doorbell. ThanksWhich could be all of them. E.g. with a 4 byte offset, we are talking 1K VQs per a 4k page.Yes, so as I replied in another thread. The doorbell of control vq should not share page with other doorbells.Except page size can be as big as 64k on some systems. The best thing is really if device just allows driver to write anywhere within the page, taking VQ number from the data.Just to make sure I understand, then there's no way to map them to guest? ThanksThen you can map them at any offset.Two more questions. 1) If the doorbell of control vq shares 64K pages with other doorbell, then we can't intercept the control vq doorbell by software.And if hardware requires doorbell e.g. at offset 0x10 in the page, we can't migrate to a device which needs it at offset 0x0.
Yes, so consider the current live migration protocol can't support such negotiation. Does this mean we need to export the offset somewhere to management? E.g sysfs or netlink?
2) Each VF should occupy at least 128K address space without the the help of flexible notification proposed by Vitaly. Is this too much if we have several thousands of vDPA instances? ThanksI guess ctrl VQs could share a page ...
Yes, the only requirement is then the doorbells of non control vqs should exclusively own one or more pages.
Doorbell of control vq could share page with reigsters except for the doobells of non control vqs.
Or there could be even no control vq implemented in the hardware. VDPA can mediate between a software control vq and vendor specific hardware control path.
I guess hypervisor can have control over this so that we can just use 4K for the common x86/ARM case.