Re: [virtio-dev] Memory sharing device

[Frank Yang <lfy@google.com>]

Revised the spec to clean up a straggling mention of callbacks, and added a concrete example of how it would be used for a video codec.

Inlined here:

\subsection{Example Use Case}\label{sec:Device Types / Host Memory Device / Example Use Case}

Suppose the guest wants to decode a compressed video buffer.

\begin{enumerate}

\item Guest creates an instance for the codec vendor id / device id / revision.

\item Guest allocates into the PCI region via config virtqueue messages.

\item Guest sends a message over the ping virtqueue for the host to back that memory.

\item Host codec device implementation exposes codec library's buffer directly to guest.

\item Guest: now that the memory is host backed, the guest mmap()'s and

  downloads the compressed video stream directly to the host buffer.

\item Guest: After a packet of compressed video stream is downloaded to the

  buffer, another message, like a doorbell, is sent on the ping virtqueue to

    consume existing compressed data. The ping message's offset field is

    set to the proper offset into the shared-mem object.

\item Host: The ping message arrives on the host and the offset is resolved to

  a physical address and then, if possible, the physical address to a host

    pointer. Since the memory is now backed, the host pointer is also

    resolved.

\item Host: Codec implementation decodes the video and puts the decoded frames

  to either a host-side display library (thus with no further guest

    communication necessary), or puts the raw decompressed frame to a

    further offset in the host buffer that the guest knows about.

\item Guest: Continue downloading video streams and hitting the doorbell, or

  optionally, wait until the host is done first. If scheduling is not that

    big of an impact, this can be done without even any further VM exit, by

    the host writing to an agreed memory location when decoding is done,

    then the guest uses a polling sleep(N) where N is the correctly tuned

    timeout such that only a few poll spins are necessary.

\item Guest: Or, the host can send back on the event virtqueue \field{revents}

  and the guest can perform a blocking read() for it.

\end{enumerate}

The unique / interesting aspects of virtio-hostmem are demonstrated:

\begin{enumerate}

\item During instance creation the host was allowed to reject the request if

  the codec device did not exist on host.

\item The host can expose a codec library buffer directly to the guest, allowing the guest to write into it with zero copy and the host to decompress again without copying.

\item Large bidirectional transfers are possible with zero copy.

\item Large bidirectional transfers are possible without scatterlists, because

  the memory is always physically contiguous.

\item It is not necessary to use socket datagrams or data streams to

  communicate the ping messages; they can be raw structs fresh off the

    virtqueue.

\item After decoding, the guest has the option but not the requirement to wait

  for the host round trip, allowing for async operation of the codec.

\item The guest has the option but not the requirement to wait for the host

  round trip, allowing for async operation of the codec.

\end{enumerate}

https://github.com/741g/virtio-spec/blob/61c500d5585552658a7c98ef788a625ffe1e201c/virtio-hostmem.tex

On Wed, Feb 13, 2019 at 10:18 AM Frank Yang <lfy@google.com> wrote:

Attached is another spec (and at https://github.com/741g/virtio-spec/commit/206b9386d76f2ce18000dfc2b218375e423ac8e0)

that tries to be more like just another virtio-pci device, but for the express purpose of controlling host memory.

Changes:

Name is now virtio-hostmem.

Explicitly uses virtio-pci transport.

Removed explicit device enumeration operation and made it part of the configuration data; the config virtqueue now mainly captures the concepts of instances and host memory allocationÂ+ sharing.

Removed mentions of triggering callbacks as that is device implementation specific and out of scope.

What it ends up being: promote host memory sharing to a device type, the purpose is guest/host communication but for when sockets are not suitable. Control messages are added. It implicitly expects that the "reversal" for virtio-pci buffers as host backed is already available (IIUC that is what virtio-fs with DAX relies on as well), and builds a thin layer on top.

Michael, is this a bit closer to what you were thinking?

On Tue, Feb 12, 2019 at 8:59 PM Frank Yang <lfy@google.com> wrote:

On Tue, Feb 12, 2019 at 8:19 PM Michael S. Tsirkin <mst@redhat.com> wrote:

On Tue, Feb 12, 2019 at 11:02:19PM -0500, Michael S. Tsirkin wrote:
> On Tue, Feb 12, 2019 at 06:50:29PM -0800, Frank Yang wrote:
> >
> >
> > On Tue, Feb 12, 2019 at 11:06 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> >Â Â ÂOn Tue, Feb 12, 2019 at 09:26:10AM -0800, Frank Yang wrote:
> >Â Â Â> BTW, the other unique aspect is that the ping messages allow a _host_
> >Â Â Âpointer
> >Â Â Â> to serve as the lump of shared memory;
> >Â Â Â> then there is no need to track buffers in the guest kernel and the device
> >Â Â Â> implementation can perform specialize buffer space management.
> >Â Â Â> Because it is also host pointer shared memory, it is also physically
> >Â Â Âcontiguous
> >Â Â Â> and there is no scatterlist needed to process the traffic.
> >
> >Â Â ÂYes at the moment virtio descriptors all pass addresses guest to host.
> >
> >Â Â ÂAbility to reverse that was part of the vhost-pci proposal a while ago.
> >Â Â ÂBTW that also at least originally had ability to tunnel
> >Â Â Âmultiple devices over a single connection.
> >
> >
> >
> > Can there be a similar proposal for virtio-pci without vhsot?
> >
> >Â Â ÂThere was nothing wrong with the proposals I think, they
> >Â Â Âjust had to be polished a bit before making it into the spec.
> >Â Â ÂAnd that runneling was dropped but I think it can be brought back
> >Â Â Âif desired, we just didn't see a use for it.
> >
> >
> > Thinking about it more, I think vhost-pci might be too much for us due to the
> > vhost requirement (sockets and IPC while we desire a highly process local
> > solution)
>
> I agree because the patches try to document a bunch of stuff.
> But I really just mean taking the host/guest interface
> part from there.

Tomorrow I'll try to write up a little bit more about the vhost pci
ideas. The patches on list go deep into envisioned implementation
detail within qemu instead of the actual host/guest interface.
You should then be able to figure out they are relevant for you.

Ok, looking forward! It does seems like a lot of what you are pointing out, and in the vhost pci, what is specified is not strictly dependent on the qemu implementation. Hopefully we can make the minimal set of changes to support our constraints.

Â

--
MST

Attachment: virtio-v1.1-wd01.pdf
Description: Adobe PDF document