Re: [virtio-dev] Re: [PATCH 4/6] vhost_vdpa: support doorbell mapping via mmap

[Jason Wang <jasowang@redhat.com>]

On 2020/5/30 äå2:30, Rob Miller wrote:
> Given the need for 4K doorbell such that QEMU can easily map, ect, and 
> assuming that I have a HW device which exposes 2 VQ's, with a 
> notification area off of BAR3, offset=whatever, notifier_multiplier=4, 
> we don't need to have 2 x 4K pages mapped into the VM for both 
> doorbells do we? The guest driver would ring DB0 at BAR4+offset, and 
> DB1 at BAR4+offset+(4*1).


This requires qemu to advertise notifier_multiplier = 4 to guest, it has 
several implications:

- guest may think control virtqueue sit in the same page then we can't 
trap access to control virtqueue, qemu may lose the state of the device
- it requires the destination to have exact the same queue_notify_off 
and notifier_multiplier in order to let migration to work

So for doorbell mapping itself, we can support the mapping of doorbells 
at once which may occupy one or more pages, but it may bring troubles 
for other features.


> The 4K per DB is usefulÂhow? This allows for QEMU trapping of 
> individual DBs, that can then be used to do what, just forward the DBs 
> via some other scheme - this makes sense for non-HW related Virtio 
> devices I guess. Is this why there is a qemu option?


The page per vq option provides extra flexibility. Note that it's the 
layout seen by guest, so actually for hardware vendor, the most easy way 
is to:

- use a single doorbell register for all virtqueues except for the 
control virtqueue (driver can differ queue index by the value wrote by 
the driver)
- do not share the page with other registers

In this way, it doesn't require much BAR space and we can still:

- map doorbell separately to guest: guest may see a page per vq doorbell 
layout but in fact all those pages are mapped into the same page
- trap the control virtqueue, (don't do mmap for control vq)

Thanks


> Rob Miller
> rob.miller@broadcom.com <mailto:rob.miller@broadcom.com>
> (919)721-3339
>
>
> On Fri, May 29, 2020 at 4:03 AM Jason Wang <jasowang@redhat.com 
> <mailto:jasowang@redhat.com>> wrote:
>
>     Currently the doorbell is relayed via eventfd which may have
>     significant overhead because of the cost of vmexits or syscall. This
>     patch introduces mmap() based doorbell mapping which can eliminate the
>     overhead caused by vmexit or syscall.
>
>     To ease the userspace modeling of the doorbell layout (usually
>     virtio-pci), this patch starts from a doorbell per page
>     model. Vhost-vdpa only support the hardware doorbell that sit at the
>     boundary of a page and does not share the page with other registers.
>
>     Doorbell of each virtqueue must be mapped separately, pgoff is the
>     index of the virtqueue. This allows userspace to map a subset of the
>     doorbell which may be useful for the implementation of software
>     assisted virtqueue (control vq) in the future.
>
>     Signed-off-by: Jason Wang <jasowang@redhat.com
>     <mailto:jasowang@redhat.com>>
>     ---
>     Âdrivers/vhost/vdpa.c | 59
>     ++++++++++++++++++++++++++++++++++++++++++++
>     Â1 file changed, 59 insertions(+)
>
>     diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c
>     index 6ff72289f488..bbe23cea139a 100644
>     --- a/drivers/vhost/vdpa.c
>     +++ b/drivers/vhost/vdpa.c
>     @@ -15,6 +15,7 @@
>     Â#include <linux/module.h>
>     Â#include <linux/cdev.h>
>     Â#include <linux/device.h>
>     +#include <linux/mm.h>
>     Â#include <linux/iommu.h>
>     Â#include <linux/uuid.h>
>     Â#include <linux/vdpa.h>
>     @@ -741,12 +742,70 @@ static int vhost_vdpa_release(struct inode
>     *inode, struct file *filep)
>     Â Â Â Â return 0;
>     Â}
>
>     +static vm_fault_t vhost_vdpa_fault(struct vm_fault *vmf)
>     +{
>     +Â Â Â Âstruct vhost_vdpa *v = vmf->vma->vm_file->private_data;
>     +Â Â Â Âstruct vdpa_device *vdpa = v->vdpa;
>     +Â Â Â Âconst struct vdpa_config_ops *ops = vdpa->config;
>     +Â Â Â Âstruct vdpa_notification_area notify;
>     +Â Â Â Âstruct vm_area_struct *vma = vmf->vma;
>     +Â Â Â Âu16 index = vma->vm_pgoff;
>     +
>     +Â Â Â Ânotify = ops->get_vq_notification(vdpa, index);
>     +
>     +Â Â Â Âvma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
>     +Â Â Â Âif (remap_pfn_range(vma, vmf->address & PAGE_MASK,
>     +Â Â Â Â Â Â Â Â Â Â Â Â Â Ânotify.addr >> PAGE_SHIFT, PAGE_SIZE,
>     +Â Â Â Â Â Â Â Â Â Â Â Â Â Âvma->vm_page_prot))
>     +Â Â Â Â Â Â Â Âreturn VM_FAULT_SIGBUS;
>     +
>     +Â Â Â Âreturn VM_FAULT_NOPAGE;
>     +}
>     +
>     +static const struct vm_operations_struct vhost_vdpa_vm_ops = {
>     +Â Â Â Â.fault = vhost_vdpa_fault,
>     +};
>     +
>     +static int vhost_vdpa_mmap(struct file *file, struct
>     vm_area_struct *vma)
>     +{
>     +Â Â Â Âstruct vhost_vdpa *v = vma->vm_file->private_data;
>     +Â Â Â Âstruct vdpa_device *vdpa = v->vdpa;
>     +Â Â Â Âconst struct vdpa_config_ops *ops = vdpa->config;
>     +Â Â Â Âstruct vdpa_notification_area notify;
>     +Â Â Â Âint index = vma->vm_pgoff;
>     +
>     +Â Â Â Âif (vma->vm_end - vma->vm_start != PAGE_SIZE)
>     +Â Â Â Â Â Â Â Âreturn -EINVAL;
>     +Â Â Â Âif ((vma->vm_flags & VM_SHARED) == 0)
>     +Â Â Â Â Â Â Â Âreturn -EINVAL;
>     +Â Â Â Âif (vma->vm_flags & VM_READ)
>     +Â Â Â Â Â Â Â Âreturn -EINVAL;
>     +Â Â Â Âif (index > 65535)
>     +Â Â Â Â Â Â Â Âreturn -EINVAL;
>     +Â Â Â Âif (!ops->get_vq_notification)
>     +Â Â Â Â Â Â Â Âreturn -ENOTSUPP;
>     +
>     +Â Â Â Â/* To be safe and easily modelled by userspace, We only
>     +Â Â Â Â * support the doorbell which sits on the page boundary and
>     +Â Â Â Â * does not share the page with other registers.
>     +Â Â Â Â */
>     +Â Â Â Ânotify = ops->get_vq_notification(vdpa, index);
>     +Â Â Â Âif (notify.addr & (PAGE_SIZE - 1))
>     +Â Â Â Â Â Â Â Âreturn -EINVAL;
>     +Â Â Â Âif (vma->vm_end - vma->vm_start != notify.size)
>     +Â Â Â Â Â Â Â Âreturn -ENOTSUPP;
>     +
>     +Â Â Â Âvma->vm_ops = &vhost_vdpa_vm_ops;
>     +Â Â Â Âreturn 0;
>     +}
>     +
>     Âstatic const struct file_operations vhost_vdpa_fops = {
>         .owner     = THIS_MODULE,
>         .open     Â= vhost_vdpa_open,
>         .release    = vhost_vdpa_release,
>         .write_iter  Â= vhost_vdpa_chr_write_iter,
>     Â Â Â Â .unlocked_ioctl = vhost_vdpa_unlocked_ioctl,
>     +   Â.mmap     Â= vhost_vdpa_mmap,
>         .compat_ioctl Â= compat_ptr_ioctl,
>     Â};
>
>     -- 
>     2.20.1