Re: [virtio-dev] Memory sharing device

[Frank Yang <lfy@google.com>]

On Tue, Feb 12, 2019 at 9:20 AM Frank Yang <lfy@google.com> wrote:

On Tue, Feb 12, 2019 at 8:46 AM Dr. David Alan Gilbert <dgilbert@redhat.com> wrote:

* Frank Yang (lfy@google.com) wrote:
> Thanks Roman for the reply. Yes, we need sensors, sound, codecs, etc. as
> well.
>
> For general string passing, yes, perhaps virtio-vsock can be used. However,
> I have some concerns about virtio-serial and virtio-vsock (mentioned
> elsewhere in the thread in rely to Stefan's similar comments) around socket
> API specialization.
>
> Stepping back to standardization and portability concerns, it is also not
> necessarily desirable to use general pipes to do what we want, because even
> though that device exists and is part of the spec already, that results in
> _de-facto_ non-portability. If we had some kind of spec to enumerate such
> 'user-defined' devices, at least we can have _de-jure_ non-portability; an
> enumerated device doesn't work as advertised.
>
> virtio-gpu: we have concerns around its specialization to virgl and
> de-facto gallium-based protocol, while we tend to favor API forwarding due
> to its debuggability and flexibility. We may use virtio-gpu in the future
> if/when it provides that general "send api data" capability.]
>
> In any case, I now have a very rough version of the spec in mind (attached
> as a patch and as a pdf).

Some thoughts (and remember I'm fairly new to virtio):

Thanks for the feedback! These are good points.

Â

 a) Please don't call it virito-user - we have vhost-user as one of the
implementations of virtio and that would just get confusing (especially
when we have a vhost-user-user implementation)

Â

Ok, the name isn't really important to me. virto-meta perhaps?
Â

 b) Your ping and event queues confuse me - they seem to be
reimplementing exactly what virtio-queues already are; aren't virtio
queues already lumps of shared memory with a 'kick' mechanism to wake up
the other end when something interesting happens?

Right, it does expose that aspect of virtqueues, but there is a unique part; it adds on a device-specific component

(the instance handle and metadata) to dispatch to and trigger the device-specific callback when it comes on the host.

I wonder if there's a way to define just the instance handle/metadata/callback parts, with other virtqueue stuff being standard.

BTW, the other unique aspect is that the ping messages allow a _host_ pointer to serve as the lump of shared memory;

then there is no need to track buffers in the guest kernel and the device implementation can perform specialize buffer space management.

Because it is also host pointer shared memory, it is also physically contiguous and there is no scatterlist needed to process the traffic.

Â

Â

 c) I think you actually have two separate types of devices that
should be treated differently;
  Â1) your high bandwidth gpu/codec
  Â2) Low bandwidth batteries/sensorsÂ

Â

Yeah, perhaps there's some way to have general virtio devices for different bandwidth levels.

Note that latency can also be an important consideration; while sensors don't send much data per second,

the time of their arrival can matter a lot. Sensors themselves can "ping" at 10^2 Hz or even higher.

 I can imagine you having a total of two device definitions and drivers
for (1) and (2).Â

 (2) feels like it's pretty similar to a socket/pipe/serial - but it
needs a way to enumerate the sensors you have, their ranges etc and a
defined format for transmitting the data. I'm not sure if it's possible
to take one of the existing socket/pipe/serial things and layer on top
of it. (is there any HID like standard for sensors like that?)

I'm not sure about there being an existing standard

(and it's possible to send different kinds of sensor / battery signals;

i.e., a fake source for testing purposes, raw sensor output from the actual hardware,

and an integrated virtual sensor that passes through derived pose information.

There's a similar breakdown when one is considering other devices such as battery and camera.)

Though I have reservations about using actual sockets/pipes to communicate,

for compatibility and latency reasons.

As another example, consider camera / video playback. I've heard that the way to approach this

is to build v4l2 and build a virtio-v4l2 layer underneath. However, this would mean either assuming v4l2 on host

which is not portable to non-Linux, or requiring additional translation layers between v4l2 in the guest and

$library on the host. With the proposed spec, a more general 'codec' device can be implemented,

tested on actual programs on multiple guests/hosts easily, and then, if it makes sense,

"promoted" to a new "virtio-codec" device type.

I think there's a central tension here that is process-related and is similar to the reasons userspace drivers are a thing in Linux;

There's a set of existing guests/hosts that use virtio on some sort of development schedule that

doesn't match the pace of development in kernels supporting virtio / the virtio spec itself.

In Virtio, we have a centralized definition of drivers and devices that is set in the middle layers

between chrome on top of VMMs and guest userspace: the guest kernel and core VMM code.

When developing for programs such as the Android Emulator,

we aim most of our changes at the guest userspace and the chrome on top of the VMM,

but it can often involve new forms of guest/host communication.

It's possible that what's being proposed also serves as a kind of L2 cache (in the form of such a virtio meta device) to deal with changes

in third-party programs like the Android Emulator with servicing the cache miss being standardization out of the meta device and into an existing concrete virtio device type.

Seen from this perspective, it can be beneficial process-wise as a way to pull in more new devices / standardize at a faster rate, that would otherwise not have been visible.

Â

 Perhaps for (1) for your GPU stuff, maybe a single virtio device
would work, with a small number of shared memory arenas but multiple
virtio queues; each (set of) queues would represent a subdevice (say
a bunch of queues for the GPU another bunch for the CODEC etc).

Well, if one device definition works well with both bandwidth regimes and has better latency properties,

I like having the one definition better.

Along these lines, I would also be open to not putting up a new standard,

and revising virtio-vsock to incorporate all the unique parts of the features proposed:

dropping socket API requirement from guest/host,

device enumeration, operation via ping+shared memory,

and callback-based operation with device definitions in the shared libraries.

The way we're approaching GPU, with the workloads we tend to run

(just UI drawing, or in 3d apps, not too many draw calls per second / limited by host GPU doing work),

still having one virtqueue seems fine as it's only used for kick

(API call parameters themselves would also live in shared memory),

though there are probably workloads that benefit from having one virtqueue per "instance".

Â

Dave

> The part of the intro in there that is relevant to the current thread:
>
> """
> Note that virtio-serial/virtio-vsock is not considered because they do not
> standardize the set of devices that operate on top of them, but in practice,
> are often used for fully general devices. Spec-wise, this is not a great
> situation because we would still have potentially non portable device
> implementations where there is no standard mechanism to determine whether or
> not things are portable. virtio-user provides a device enumeration
> mechanism
> to better control this.
>
> In addition, for performance considerations in applications such as graphics
> and media, virtio-serial/virtio-vsock have the overhead of sending actual
> traffic through the virtqueue, while an approach based on shared memory can
> result in having fewer copies and virtqueue messages. virtio-serial is also
> limited in being specialized for console forwarding and having a cap on the
> number of clients. virtio-vsock is also not optimal in its choice of
> sockets
> API for transport; shared memory cannot be used, arbitrary strings can be
> passed without an designation of the device/driver being run de-facto, and
> the
> guest must have additional machinery to handle socket APIs. In addition, on
> the host, sockets are only dependable on Linux, with less predictable
> behavior
> from Windows/macOS regarding Unix sockets. Waiting for socket traffic on
> the
> host also requires a poll() loop, which is suboptimal for latency. With
> virtio-user, only the bare set of standard driver calls
> (open/close/ioctl/mmap/read) is needed, and RAM is a more universal
> transport
> abstraction. We also explicitly spec out callbacks on host that are
> triggered
> by virtqueue messages, which results in lower latency and makes it easy to
> dispatch to a particular device implementation without polling.
>
> """
>
> On Tue, Feb 12, 2019 at 6:03 AM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> > On Tue, Feb 12, 2019 at 02:47:41PM +0100, Cornelia Huck wrote:
> > > On Tue, 12 Feb 2019 11:25:47 +0000
> > > "Dr. David Alan Gilbert" <dgilbert@redhat.com> wrote:
> > >
> > > > * Roman Kiryanov (rkir@google.com) wrote:
> > > > > > > Our long term goal is to have as few kernel drivers as possible
> > and to move
> > > > > > > "drivers" into userspace. If we go with the virtqueues, is there
> > > > > > > general a purpose
> > > > > > > device/driver to talk between our host and guest to support
> > custom hardware
> > > > > > > (with own blobs)?
> > > > > >
> > > > > > The challenge is to answer the following question:
> > > > > > how to do this without losing the benefits of standartization?
> > > > >
> > > > > We looked into UIO and it still requires some kernel driver to tell
> > > > > where the device is, it also has limitations on sharing a device
> > > > > between processes. The benefit of standardization could be in
> > avoiding
> > > > > everybody writing their own UIO drivers for virtual devices.
> > > > >
> > > > > Our emulator uses a battery, sound, accelerometer and more. We need
> > to
> > > > > support all of this. I looked into the spec, "5 Device types", and
> > > > > seems "battery" is not there. We can invent our own drivers but we
> > see
> > > > > having one flexible driver is a better idea.
> > > >
> > > > Can you group these devices together at all in their requirements?
> > > > For example, battery and accelerometers (to me) sound like
> > low-bandwidth
> > > > 'sensors' with a set of key,value pairs that update occasionally
> > > > and a limited (no?) amount of control from the VM->host.
> > > > A 'virtio-values' device that carried a string list of keys that it
> > > > supported might make sense and be enough for at least two of your
> > > > device types.
> > >
> > > Maybe not a 'virtio-values' device -- but a 'virtio-sensors' device
> > > looks focused enough without being too inflexible. It can easily
> > > advertise its type (battery, etc.) and therefore avoid the mismatch
> > > problem that a too loosely defined device would be susceptible to.
> >
> > Isn't virtio-vsock/vhost-vsock a good fit for this kind of general
> > string passing? People seem to use it exactly for this.
> >
> > > > > Yes, I realize that a guest could think it is using the same device
> > as
> > > > > the host advertised (because strings matched) while it is not. We
> > > > > control both the host and the guest and we can live with this.
> > >
> > > The problem is that this is not true for the general case if you have a
> > > standardized device type. It must be possible in theory to switch to an
> > > alternative implementation of the device or the driver, as long as they
> > > conform to the spec. I think a more concretely specified device type
> > > (like the suggested virtio-values or virtio-sensors) is needed for that.
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> > > For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
> >



--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK