[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]
Subject: Re: [PATCH v2 0/6] Extend vhost-user to support VFIO based accelerators
On Thu, Mar 22, 2018 at 06:40:18PM +0200, Michael S. Tsirkin wrote: > On Mon, Mar 19, 2018 at 03:15:31PM +0800, Tiwei Bie wrote: [...] > > > > Below link is an example showing how to setup a such environment > > via nested VM. In this case, the virtio device in the outer VM is > > the accelerator. It will be used to accelerate the virtio device > > in the inner VM. In reality, we could use virtio ring compatible > > hardware device as the accelerators. > > > > http://dpdk.org/ml/archives/dev/2017-December/085044.html > > I understand that it might be challenging due to > the tight coupling with VFIO. Still - isn't there > a way do make it easier to set a testing rig up? > > In particular can we avoid the dpdk requirement for testing? > If we want to try vDPA (e.g. use one virtio device to accelerate another virtio device of a VM), I think we need vDPA. Otherwise we will need to write a VFIO based userspace virtio driver and find another vhost-user backend. > > > > In above example, it doesn't require any changes to QEMU, but > > it has lower performance compared with the traditional VFIO > > based PCI passthru. And that's the problem this patch set wants > > to solve. > > > > The performance issue of vDPA/vhost-user and solutions > > ====================================================== > > > > For vhost-user backend, the critical issue in vDPA is that the > > data path performance is relatively low and some host threads are > > needed for the data path, because some necessary mechanisms are > > missing to support: > > > > 1) guest driver notifies the device directly; > > 2) device interrupts the guest directly; > > > > So this patch set does some small extensions to the vhost-user > > protocol to make both of them possible. It leverages the same > > mechanisms (e.g. EPT and Posted-Interrupt on Intel platform) as > > the PCI passthru. > > Not all platforms support posted interrupts, and EPT isn't > required for MMIO to be mapped to devices. > > It probably makes sense to separate the more portable > host notification offload from the less portable > guest notification offload. > Make sense. I'll split the two types of offloads. Thanks for the suggestion! > > > > A new protocol feature bit is added to negotiate the accelerator > > feature support. Two new slave message types are added to control > > the notify region and queue interrupt passthru for each queue. > > >From the view of vhost-user protocol design, it's very flexible. > > The passthru can be enabled/disabled for each queue individually, > > and it's possible to accelerate each queue by different devices. > > More design and implementation details can be found from the last > > patch. > > > > Difference between vDPA and PCI passthru > > ======================================== > > > > The key difference between PCI passthru and vDPA is that, in vDPA > > only the data path of the device (e.g. DMA ring, notify region and > > queue interrupt) is pass-throughed to the VM, the device control > > path (e.g. PCI configuration space and MMIO regions) is still > > defined and emulated by QEMU. > > > > The benefits of keeping virtio device emulation in QEMU compared > > with virtio device PCI passthru include (but not limit to): > > > > - consistent device interface for guest OS in the VM; > > - max flexibility on the hardware (i.e. the accelerators) design; > > - leveraging the existing virtio live-migration framework; > > > > Why extend vhost-user for vDPA > > ============================== > > > > We have already implemented various virtual switches (e.g. OVS-DPDK) > > based on vhost-user for VMs in the Cloud. They are purely software > > running on CPU cores. When we have accelerators for such NFVi applications, > > it's ideal if the applications could keep using the original interface > > (i.e. vhost-user netdev) with QEMU, and infrastructure is able to decide > > when and how to switch between CPU and accelerators within the interface. > > And the switching (i.e. switch between CPU and accelerators) can be done > > flexibly and quickly inside the applications. > > > > More details about this can be found from the Cunming's discussions on > > the RFC patch set. > > > > Update notes > > ============ > > > > IOMMU feature bit check is removed in this version, because: > > > > The IOMMU feature is negotiable, when an accelerator is used and > > it doesn't support virtual IOMMU, its driver just won't provide > > this feature bit when vhost library querying its features. And if > > it supports the virtual IOMMU, its driver can provide this feature > > bit. It's not reasonable to add this limitation in this patch set. > > Fair enough. Still: > Can hardware on intel platforms actually support IOTLB requests? > Don't you need to add support for vIOMMU shadowing instead? > For the hardware I have, I guess they can't for now. Best regards, Tiwei Bie
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]