Re: [virtio-dev] Writing a Virtio-net device in Lua (Snabb Switch)

["Michael S. Tsirkin" <mst@redhat.com>]

On Fri, Feb 14, 2014 at 03:38:05PM +0100, Luke Gorrie wrote:
> Howdy!
> 
> I'm writing a Virtio-net device in Lua. I'd love to have a little feedback to
> help me understand whether I'm on the right track with my implementation and
> what details of the spec I have misunderstood.

I wonder whether Lua provides low level memory barriers
used for virtio ring operation.
They are required for correctness on SMP systems.

> If we're lucky then maybe we can even uncover some interesting new use cases
> for Virtio-net. I hope this will in some way be interesting to other people
> too, anyway.
> 
> I and others are developing a fully zero-copy software switch for QEMU in
> userspace. It's called Snabb Switch and it's for ISPs who have to shuffle
> millions of packets per second per box.
> 
> The zero-copy design works by creating a 1:1 mapping between Virtio-net queues
> and Intel VMDq hardware queues in the NIC. This means that I/O is performed by
> simply translating descriptors between Virtio-net and hardware-native formats
> without touching the packet data.
> 
> The QEMU interface is based on vhost_user (link below), a userspace clone of
> the Linux vhost-net kernel interface. That means the switch handles the vrings
> but not the PCI level.
> 
> I want to be able to hack packets "in flight" before making them available with
> translated descriptors. For example, to be able to perform encap/decap of
> tunnel headers. I'm aiming to minimize the number of bytes that have to be
> touched in these cases, e.g. I would like to be able to isolate the packet
> headers into a small iovec that's cheap to rewrite and keep the rest of the
> packet data in larger iovecs afterwards.
> 
> Here are the questions on my mind:
> 
> Are MERGEABLE descriptors a good choice for me? This triggers the
> 'mergeable_rx_bufs' behavior in the Linux driver and provides me with 4KB
> buffers.

With recent drivers, you might get smaller buffers as well,
starting with
Commit 2613af0ed18a ("virtio_net: migrate mergeable rx buffers to page frag
allocators")

See also
virtio-net: auto-tune mergeable rx buffer size for improved
performance

which changes the size dynamically.


> I believe that I can send packets to VMs using multiple buffers, and
> that I can say how many bytes of each buffer should be used, but cannot use
> offsets (data must start at the beginning of the buffer).

Yes.

> Are INDIRECT descriptors a good choice for me? I don't use it today and I don't
> know if it would bring any advantages. If it would in practice give me more
> buffers from the VM then that would be positive. (I seem to only get 256
> buffers from qemu.)

In our testing it's an advantage for guest to external but not
external to guest.

> What glaring mistakes have I made in my code based on misunderstandings of
> Virtio-net? I'm sure there are several!
> 
> Any advice appreciated, and thank you for reading this far. :-)
> 
> vhost_user design: http://www.virtualopensystems.com/en/solutions/guides/
> snabbswitch-qemu/
> My virtio code: https://github.com/SnabbCo/snabbswitch/blob/vhostuser_v3/src/
> apps/vhost/vhost_user.lua
> Project: https://github.com/SnabbCo/snabbswitch/wiki/
> 
> Cheers,
> -Luke
> 
>