Virtio-loopback: PoC of a new Hardware Abstraction Layer for non-Hypervisor environments based on Virtio

[Timos Ampelikiotis <t.ampelikiotis@virtualopensystems.com>]

Dear virtio-dev community,

I would like to introduce you Virtio-loopback, Proof of Concept (PoC) that
we have been working on at Virtual Open Systems in the context of the
Automotive Grade Linux community (Virtualization & Containers expert

group - EG-VIRT).

We consider this work as a PoC and we are not currently planning
upstream. However, if the zero-copy or any other aspect of this work
is interesting for other Virtio implementations, we would be glad to
discuss more.

Overview:
---------

Virtio-loopback is a new hardware abstraction layer designed for non-Hypervisor
environments based on virtio. The main objective is to enable applications
communication with vhost-user devices in a non-hypervisor environment.

More in details, Virtio-loopback's design consists of a new transport
(Virtio-loopback), a user-space daemon (Adapter), and a vhost-user device.
The data path has been implemented using the "zero-copy" principle, where
vhost-user devices access virtqueues directly in the kernel space. This first
implementation supports multi-queues, does not require virtio-protocol changes
and applies minor modifications to the vhost-user library. Supported vhost-user
devices are today vhost-user-rng (both rust and C version), vhost-user-input
and vhost-user-blk.

Motivation & requirements:
-------------------------

1. Enable the usage of the same user-space driver on both virtualized and
 Ânon-virtualized environments.

2. Maximize performance with zero copy design principles

3. Applications using such drivers are unchanged and transparently running in
 Âboth virtualized or non-virtualized environment.

Design description:
-------------------

a) Component's description:
--------------------------

The Virtio-loopback architecture consists of three main components described below:

1) Driver: In order to route the VIRTIO communication in user-space
 Âvirtio-loopback driver was implemented and consists of:
 Â- A new transport layer which is based on virtio-mmio and it is
  Âresponsible of routing the read/write communication of the virtio
  Âdevice to the adapter binary.
 Â- A character device which works as an intermediate layer between the
  Âuser-space components and the transport layer. The character device helps
  Âthe adapter to provide all the required information and initialize the
  Âtransport and at the same time, provides direct access to the vrings
  Âfrom user-space. The access to the vrings is based on a memory mapping
  Âmechanism which gives the ability to the vhost-user device to read and
  Âwrite data directly into kernel's memory without the need of any copy.

2) Adapter: Implements the role that QEMU had in the corresponding virtualized
 Âscenario. Specifically, combines the functionality of two main QEMU
 Âcomponents, virtio-mmio transport emulation and vhost-user backend, in order
 Âto work as a bridge between the transport and the vhost-user device. The two
 Âmain parts of the adapter are:
 Â- A vhost-user backend which is the main communication point with the
  Âvhost-user device.
 Â- A virtio-emulation which handles mostly the messages coming from the
  Âdriver and translates them into vhost-user messages/actions.

3) Vhost-user device: This components required only minimal modifications to
 Âmake the vrings directly accessible in kernel's memory.

b) Communication between the virtio-loopback components:
-------------------------------------------------------

After the describing the role of its component, a few details need to be given
about how they interact with each other and the mechanisms used.

1) Transport & Adapter:
 Â- The two components share a communication data structure which describes
  Âthe current read/write operation requested by the transport.
 Â- When this data structure has been filled with all the required information
  Âthe transport triggers and EventFD and waits. The adapter wakes up, takes
  Âthe corresponding actions and finally notifies and unlocks the transport
  Âby calling an IOCTL system call.
 Â- Compared to the virtualized environment scenario, the adapter calls an
  ÂIOCTL system call to the driver in place of an interrupt.

2) Adapter & Vhost-user device:
 Â- The mechanisms used between these two component are the same as in
  Âthe virtualized environment case.
  Âa) A UNIX socket is in place to exchange any VHOST-USER messages.
  Âb) EventFDs are being used in order to trigger VIRTIO kick/call
    requests.

3) Transport & Vhost-user device:
 Â- Since the Vrings are allocated into the Kernel's memory, vhost-user
  Âdevice needs to communicate and request access from the virtio-loopback
  Âdriver. These requirement is served by implementing MMAP and IOCTL system
  Âcalls in the driver.

c) Vrings & Zero copy memory mapping mechanism:
----------------------------------------------

The vrings are allocated by the virtio driver into the kernel's memory space
and in order to be accessible by the user-space, especially by the vhost-user
device, a new memory mapping mechanism needed to be created into the
virtio-loopback driver. The new memory mapping mechanism is based on a
page-fault handler which maps the accessed pages on-demand.

Known issues & room for improvement:
-----------------------------------

Known limitation found in the current implementation:
- The memory mapping mechanism needs improvements, in the current
 implementation the device can potentially access the whole kernel's
 memory. A more fine grained mmapping can be set by the kernel by
 narrowing down the memory block shared.

Possible next development targets might be about:
- Security checks for the memory shared with the user-space (vhost user-device)
- Add parallel device handling for the virtio-loopback transport and adapter
- Add support for more vhost-user devices

More information:
----------------

The full description of the technology can be found in the links below:
-ÂVirtio-loopback design document
-ÂHow to test the technology

Links for all the key components of the design can be found below:
1)ÂVirtio-loopback-transport
2)ÂAdapter
3)ÂVhost-user devices in Qemu

Virtio-loopback has been tested on RCAR-M3 board (AGL needlefish) and x86
systems (Fedora 37). The results have been found to be comparable with VDUSE
technology in virtio-blk case:
-ÂAutomotive Grade Linux All Member Meeting Spring (8-9/03/2023) - Presentation

 + Activity done in the context of the AERO EU project (grant agreement No 101092850)

Thank you for taking the time to review this PoC,
I would appreciate your feedback and suggestions for improvements.

Best regards,
Timos Ampelikiotis