Re: [virtio-dev] Vhost-pci RFC2.0

[Wei Wang <wei.w.wang@intel.com>]

On 05/02/2017 08:48 PM, Stefan Hajnoczi wrote:
> On Thu, Apr 20, 2017 at 01:51:24PM +0800, Wei Wang wrote:
> > On 04/19/2017 11:24 PM, Stefan Hajnoczi wrote:
> > > On Wed, Apr 19, 2017 at 11:42 AM, Wei Wang <wei.w.wang@intel.com> wrote:
> > > > On 04/19/2017 05:57 PM, Stefan Hajnoczi wrote:
> > > > > On Wed, Apr 19, 2017 at 06:38:11AM +0000, Wang, Wei W wrote:
> > > > > > We made some design changes to the original vhost-pci design, and want to
> > > > > > open
> > > > > > a discussion about the latest design (labelled 2.0) and its extension
> > > > > > (2.1).
> > > > > > 2.0 design: One VM shares the entire memory of another VM
> > > > > > 2.1 design: One VM uses an intermediate memory shared with another VM for
> > > > > >                         packet transmission.
> > > > > Hi,
> > > > > Can you talk a bit about the motivation for the 2.x design and major
> > > > > changes compared to 1.x?
> > > > 1.x refers to the design we presented at KVM Form before. The major
> > > > change includes:
> > > > 1) inter-VM notification support
> > > > 2) TX engine and RX engine, which is the structure built in the driver. From
> > > > the device point of view, the local rings of the engines need to be
> > > > registered.
> > > It would be great to support any virtio device type.
> > Yes, the current design already supports the creation of devices of
> > different types.
> > The support is added to the vhost-user protocol and the vhost-user slave.
> > Once the slave handler receives the request to create the device (with
> > the specified device type), the remaining process (e.g. device realize)
> > is device specific.
> > This part remains the same as presented before
> > (i.e.Page 12 @ http://www.linux-kvm.org/images/5/55/02x07A-Wei_Wang-Design_of-Vhost-pci.pdf).
> > > The use case I'm thinking of is networking and storage appliances in
> > > cloud environments (e.g. OpenStack).  vhost-user doesn't fit nicely
> > > because users may not be allowed to run host userspace processes.  VMs
> > > are first-class objects in compute clouds.  It would be natural to
> > > deploy networking and storage appliances as VMs using vhost-pci.
> > >
> > > In order to achieve this vhost-pci needs to be a virtio transport and
> > > not a virtio-net-specific PCI device.  It would extend the VIRTIO 1.x
> > > spec alongside virtio-pci, virtio-mmio, and virtio-ccw.
> > Actually it is designed as a device under virtio-pci transport. I'm
> > not sure about the value of having a new transport.
> >
> > > When you say TX and RX I'm not sure if the design only supports
> > > virtio-net devices?
> > Current design focuses on the vhost-pci-net device. That's the
> > reason that we have TX/RX here. As mention above, when the
> > slave invokes the device creation function, the execution
> > goes to each device specific code.
> >
> > The TX/RX is the design after the device creation, so it is specific
> > to vhost-pci-net. For the future vhost-pci-blk, that design can
> > have its own request queue.
> Here is my understanding based on your vhost-pci GitHub repo:
>
> VM1 sees a normal virtio-net-pci device.  VM1 QEMU is invoked with a
> vhost-user netdev.
>
> VM2 sees a hotplugged vhost-pci-net virtio-pci device once VM1
> initializes the device and a message is sent over vhost-user.

Right.

> There is no integration with Linux drivers/vhost/ code for VM2.  Instead
> you are writing a 3rd virtio-net driver specifically for vhost-pci.  Not
> sure if it's possible to reuse drivers/vhost/ cleanly but that would be
> nicer than implementing virtio-net again.

vhost-pci-net is a standalone network device with its own unique
device id, and the device itself is different from virtio-net (e.g.
different virtqueues), so I think it would be more reasonable to
let vhost-pci-net have its own driver.

There are indeed some functions in vhost-pci-net that looks
similar to those in virtio-net (e.g. try_fill_recv()). I haven't thought
of a good way to reuse them yet, because the interfaces are not
completely the same, for example, vpnet_info and virtnet_info,
which need to be passed to the functions, are different.

> Is the VM1 vhost-user netdev a normal vhost-user device or does it know
> about vhost-pci?

Share the QEMU booting commands, which would be helpful:
VM1(vhost-pci-net):
-chardev socket,id=slave1,server,wait=off,path=${PATH_SLAVE1} \
-vhost-pci-slave socket,chardev=slave1

VM2(virtio-net):
-chardev socket,id=sock2,path=${PATH_SLAVE1} \
-netdev type=vhost-user,id=net2,chardev=sock2,vhostforce \
-device virtio-net-pci,mac=52:54:00:00:00:02,netdev=net2

The netdev doesn't know about vhost_pci, but the vhost_dev knows
it via
vhost_dev->protocol_features &
    (1ULL << VHOST_USER_PROTOCOL_F_VHOST_PCI),

The vhost-pci specific messages need to be sent in the vhost-pci
case. For example, at the end of vhost_net_start(), if it detects the
slave is vhost-pci, it will send a
VHOST_USER_SET_VHOST_PCI_START message to the slave(VM1).

> It's hard to study code changes in your vhost-pci repo because
> everything (QEMU + Linux + your changes) was committed in a single
> commit.  Please keep your changes in separate commits so it's easy to
> find them.
Thanks a lot for reading the draft code. I'm working to do some
cleanup and split it into patches. I will post out the QEMU side
patches soon.


Best,
Wei