Seeking guidance for custom virtIO device

["Eftime, Petre" <epetre@amazon.com>]

Hi all,

I am looking for guidance on how to proceed with regards to either reserving a virtio device ID for a specific device for a particular usecase  or for formalizing a device type that could be potentially used by others.

We have developed a virtio device that acts as a transport for API calls between a guest userspace library and a backend server in the host system.
Our requirements are:
* multiple clients in the guest (multiple servers is not required)
* provide an in-order, reliable datagram transport mechanism
* datagram size should be either negotiable or large (16k-64k?)
* performance is not a big concern for our usecase

The reason why we used a special device and not something else is the following:
* vsock spec does not contain a datagram specification (eg. SOCK_DGRAM, SOCK_SEQPACKET) and the effort of updating the Linux driver and other implementations for this particular purposeÂÂseemed relatively high. The path to approach this problem wasn't clear. Vsock today only works in SOCK_STREAM mode and this is not ideal: the receiver must implement additional state and buffer incoming data,ÂÂadding complexity and host resource usage.
* rpmsg has some implementation limitation (eg. from what I understand, Linux can have at most 512 bytes per message) and the programming model seems complicated and doesn't necesarrily fit our needs, since it's fully callback based and would require additional complex logic to make sure it behaves well for our usecase (at least in Linux). I am also not entirely convinced you can implement endpoints that cross VM boundaries (eg. guest <-> host).
* virtio-serial is not datagram based (and therefore has similar issues to vsock in Stream mode).

For our device, we decided to:
* use a single virtio queue
* each element in the virtio queue contains two buffers: the first one is the request and the second one is the reply (similar to a virtio-block device)
* the guest driver itself doesn't need to be capable to understand the requests or replies, everything is handled in between guest userspace and host userspace
* the guest API passes both a send buffer and a receive buffer and calls are blocking, if you need concurrency you could create multiple threads
* since there is only one endpoint, there's no source or destination added to the packages.

Is there anyone working on a device that solves a similar problem to this one, which we might reuse or collaborate on? I could potentially see this type of device being useful to others, maybe with some minor adjustments.
If not, what is the path to having our device live somewhere in the virtio world? If there is no reason to include our device in the spec proper, we at least want to reserve an identifier for it, to prevent accidental collisions with other devices. I see that there are other devices which are rather specialized and not yet formalized listed in the spec, but I don't fully graps what steps I need to take in order to get an ID. Would submitting a patch with the ID be enough?

Thank you,
Petre Eftime





Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.