Re: [virtio-dev] [FYI] virtio-media published

[Alexander Gordeev <alexander.gordeev@opensynergy.com>]

Hello Alexandre,

On 08.12.23 07:46, Alexandre Courbot wrote:
> Hello everyone,
>
> This is just a heads-up to announce the initial publication of virtio-media:
>
> https://github.com/Gnurou/virtio-media
>
> The name has changed slightly, but this project is essentially the
> refinement and continuation of my virtio-v4l2 proposal. The reason for
> moving forward with this is, to be candid, the unclear direction and
> lack of progress regarding the future of virtio-video, and the
> non-consensual way in which its development has been, let's say
> "redirected".

Great! I like the new name. The old one was confusing IMO. I'm going to 
have a closer look later.

I'd be happy to clarify the current progress and the future direction of 
the virtio-video. I'd like to make these major changes in the next spec 
draft:

1. Represent the capabilities and parameters with TLVs. Basically I'd 
like to take the data format from VIDIOC_G_EXT_CTRLS/VIDIOC_S_EXT_CTRLS,
flatten it by replacing the pointers with the actual data (this would be 
then in fact a sequence of TLVs). Then use this data format when getting 
capabilities (all of the VIDIOC_ENUM* and VIDIOC_QUERY* in a single 
call), getting and setting parameters (parameters + controls in V4L2 
terms). Here I'd like to reuse the control ids and value definitions as 
much as possible. Maybe this data presentation could be used in V4L2 
later. I think it would be digestible, as it is an extension of the 
extended controls. Indeed it was not clear, I even thought device trees 
would be a great solution at some point (see [1] for the rationale), but 
several people convinced me, that's not going to be merged in the kernel.

BTW this activity was influenced by your comment in [2]:

> Beware of not repeating the same mistake as v5 (all controls under one
> big structure).

2. Make the input and output resource queues explicit by turning them 
into virtio queues. It became clear, that after making the commands non 
blocking, some of the queue control moves to these internal resource 
queues. Then I thought: why make another queue implementation, when we 
already have one? So I think it makes sense to turn them into virtio 
queues. I still would like to keep the delayed responses over the event 
queue because I like to have this ordering of the events. Otherwise the 
DRAIN has to be done in some other way. Also I remember the comment by 
BartÅomiej Grzesik in [3]:

> I remember having a few thoughts on how it could be solved and I
> think that removing the need to block those descriptors is the best
> approach in my opinion. One would argue that preallocating descriptors
> for this purpose or splitting the command queue to input, output and
> control might be a viable solution or per stream. However it would
> only delay the issue in time or could cause other streams to "starve".

So I'd like to try both approaches at once.

I hope to finish the next draft in 3-4 weeks and publish in January. 
Hopefully the virtio release will be done and folks will have some time 
for a review.

Also I've been busy updating the virtio-video driver. First I decided to 
merge all the existing patches. We have a lot of not yet published 
changes internally, also you and your team made a lot of changes to the 
driver. I went through all of these, merged everything carefully, 
learned about some of the problems, that you encountered. This was a big 
chunk of work and now it is finished, see [4]. Still working on other 
driver updates.

The first problem was about making a distinction between guest pages and 
exported buffers. The final solution in your patches AFAIU is to map 
VB2_MEMORY_USERPTR to guest pages and VB2_MEMORY_DMABUF to the exported 
objects because this is the only hint, that we have from REQBUFS. I 
think this is not good enough, because dmabufs can be of different 
types: it could be an exported object or, for example, one created with 
udmabuf. I don't think there is a way to provide this information to the 
device until the first buffer comes, right? So the spec should be 
changed in this regard. I think it would be enough to add a requirement, 
that all the buffers belonging to a queue must be of the same type.

Another problem is setting the DMA masks and other such stuff. Our 
current approach is to register the parent device, which has all the 
related fields properly set as it is done in virtio-gpu, see [5].

Overall I'd like to highlight the fact, that you and your team already 
have a lot of influence on the virtio-video direction. I was very busy 
with investigations trying to select the right solution and merging the 
patches, but I think this only makes the whole thing better. :) So your 
comments are very welcome.

> The repo above contains the specification in the README file
> (presented in a more informal way than the virtio specification), as
> well as a guest V4L2 driver able to pass v4l2-compliance when proxying
> the vivid/vicodec virtual devices or an actual UVC camera using the
> crosvm V4L2 proxy device [1]. As of now it is basically
> feature-complete and offers everything that virtio-video is supposed
> to eventually do. I am considering adding support for multiple devices
> and requests to allow more complex camera setups.
>
> Since the initial proposal has been rejected I have no intent to push
> this forward for merging in the virtio specification, so this will
> live out of the official spec. However, it is likely that we will soon
> switch to this solution for video support in ChromeOS and possibly
> other Google projects with a similar need for a stable virtualization
> solution for media devices.
>
> Anyone who is interested in using this for their project, or with
> specific requests, is welcome to contact me or open issues on the
> Github project.
>
> Cheers,
> Alex.
>
> [1] https://chromium-review.googlesource.com/c/crosvm/crosvm/+/506532

Kind regards,
Alexander

[1] 
https://lore.kernel.org/virtio-dev/a9235fe7-7448-fa9f-ea52-fd27f4845bc4@opensynergy.com/
[2] 
https://lore.kernel.org/virtio-dev/CAPBb6MXw5PebCXYBvXOP_b2j+t-1Y_rSJv7kWkLPsa3X+uM9gA@mail.gmail.com/
[3] 
https://lore.kernel.org/virtio-dev/CAKxn3ec7nDy9B0nyHUg3GfXt_oktrC91kB3QWXM-sGh8ectE5A@mail.gmail.com/
[4] https://github.com/OpenSynergy/linux/tree/virtio-video
[5] 
https://github.com/torvalds/linux/commit/b5c9ed70d1a94c59dad7b1ecfc928863c0fe6ac0

--
Alexander Gordeev
Senior Software Engineer

OpenSynergy GmbH
Rotherstr. 20, 10245 Berlin
www.opensynergy.com