Re: [virtio-dev] Re: [Qemu-devel] [virtio-dev] Re: [virtio-dev] Re: [PATCH v2 00/16] Vhost-pci for inter-VM communication

["Michael S. Tsirkin" <mst@redhat.com>]

On Thu, May 25, 2017 at 08:31:09PM +0800, Jason Wang wrote:
> 
> 
> On 2017年05月25日 20:22, Jason Wang wrote:
> > > > 
> > > > Even with vhost-pci to virito-net configuration, I think rx
> > > > zerocopy could be achieved but not implemented in your driver
> > > > (probably more easier in pmd).
> > > > 
> > > Yes, it would be easier with dpdk pmd. But I think it would not be
> > > important in the NFV use case,
> > > since the data flow goes to one direction often.
> > > 
> > > Best,
> > > Wei
> > > 
> > 
> > I would say let's don't give up on any possible performance optimization
> > now. You can do it in the future.
> > 
> > If you still want to keep the copy in both tx and rx, you'd better:
> > 
> > - measure the performance of larger packet size other than 64B
> > - consider whether or not it's a good idea to do it in vcpu thread, or
> > move it to another one(s)
> > 
> > Thanks
> 
> And what's more important, since you care NFV seriously. I would really
> suggest you to draft a pmd for vhost-pci and use it to for benchmarking.
> It's real life case. OVS dpdk is known for not optimized for kernel drivers.
> 
> Good performance number can help us to examine the correctness of both
> design and implementation.
> 
> Thanks

I think that's a very valid point. Linux isn't currently optimized to
handle packets in device BAR.

There are several issues here and you do need to address them in the
kernel, no way around that:

1. lots of drivers set protection to
        vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);

   vfio certainly does, and so I think does pci sysfs.
   You won't get good performance with this, you want to use
   a cacheable mapping.
   This needs to be addressed for pmd to work well.

2. linux mostly assumes PCI BAR isn't memory, ioremap_cache returns __iomem
   pointers which aren't supposed to be dereferenced directly.
   You want a new API that does direct remap or copy if not possible.
   Alternatively remap or fail, kind of like pci_remap_iospace.
   Maybe there's already something like that - I'm not sure.


Thanks,
MST

-- 
MST