Re: [virtio-dev] Re: Fwd: Qemu Support for Virtio Video V4L2 driver

[Keiichi Watanabe <keiichiw@chromium.org>]

Hi Saket,

On Mon, May 11, 2020 at 9:33 PM Saket Sinha <saket.sinha89@gmail.com> wrote:
>
> Hi Keiichi,
>
> > > > I do not support the approach of  QEMU implementation forwarding
> > > > requests to the host's vicodec module since  this can limit the scope
> > > > of the virtio-video device only for testing,
> > >
> > > That was my understanding as well.
> >
> > Not really because the API which the vicodec provides is V4L2 stateful
> > decoder interface [1], which are also used by other video drivers on
> > Linux.
> > The difference between vicodec and actual device drivers is that
> > vicodec performs decoding in the kernel space without using special
> > video devices. In other words, vicodec is a software decoder in kernel
> > space which provides the same interface with actual video drivers.
> > Thus, if the QEMU implementation can forward virtio-video requests to
> > vicodec, it can forward them to the actual V4L2 video decoder devices
> > as well and VM gets access to a paravirtualized video device.
> >
> > The reason why we discussed vicodec in the previous thread was it'll
> > allow us to test the virtio-video driver without hardware requirement.
> >
> > [1] https://www.kernel.org/doc/html/latest/media/uapi/v4l/dev-decoder.html
> >
>
> Thanks for clarification.
>
> Could  you provide your views if it would be possible to support also
> paravirtualized v4l-subdev devices which is enabled by media
> controller to expose ISP processing blocks to linux userspace.
> Ofcourse, we might need to change implementation and spec to support that
> Please refer (1) for details.

Again, the current virtio-video protocol and driver only support video
encoding and decoding. We had no detailed discussion about camera
supports.
Moreover, I personally disagree with supporting video capturing in
virtio-video protocol. Instead, I believe it's better to have a
separate protocol like "virtio-camera". Decoupling video codec APIs
and camera APIs should make protocols simpler and easier to maintain.
I suggested this idea in [1].

So, the answer to your question is:
No in virtio-video protocol. But, it's possible to start designing a
new "virtio-camera" protocol that supports camera features including
image processing.

[1] https://markmail.org/message/4q2g5oqniw62pmqd

>
> > >
> > > > which instead can be used with multiple use cases such as -
> > > >
> > > > 1. VM gets access to paravirtualized  camera devices which shares the
> > > > video frames input through actual HW camera attached to Host.
> > >
> > > This use-case is out of the scope of virtio-video. Initially I had a plan to
> > > support capture-only streams like camera as well, but later the decision was
> > > made upstream that camera should be implemented as separate device type. We
> > > still plan to implement a simple frame capture capability as a downstream
> > > patch though.
> > >
> > > >
> > > > 2. If Host has multiple video devices (especially in ARM SOCs over
> > > > MIPI interfaces or USB), different VM can be started or hotplugged
> > > > with selective video streams from actual HW video devices.
> > >
> > > We do support this in our device implementation. But spec in general has no
> > > requirements or instructions regarding this. And it is in fact flexible enough
> > > to provide abstraction on top of several HW devices.
> > >
> > > >
> > > > Also instead of using libraries like Gstreamer in Host userspace, they
> > > > can also be used inside the VM userspace after getting access to
> > > > paravirtualized HW camera devices .
> >
> > Regarding Gstreamer, I intended this video decoding API [2]. If QEMU
> > can translate virtio-video requests to this API, we can easily support
> > multiple platforms.
> > I'm not sure how feasible it is though, as I have no experience of
> > using this API by myself...
> >
> > [2] https://gstreamer.freedesktop.org/documentation/tutorials/playback/hardware-accelerated-video-decoding.html
> >
>
> Like pointed out above, Gstreamer is not the only framework present there.
> We have the newer libcamera framework [2] and then Openmax (used in
> Android Hal )
> Refer [3] for comparison.

It seems that we had miscommunication here. While I had mentioned
Gstreamer as a generic implementation to cover "video decoding" APIs
on various platforms, you were talking about "camera" APIs.
As I said above, virtio-video is NOT designed for cameras.

For abstraction of video decoding APIs, I don't know any better
library than Gstreamer. For cameras, libcamera sounds good, but I'm
not so familiar with this area...

Best regards,
Keiichi


>
> My intentions are to make the implementation more generic so that it
> can be used by different frameworks on different platforms.
>
> [1]: https://static.sched.com/hosted_files/osseu19/21/libcamera.pdf
> [2]: http://libcamera.org
> [3]: https://processors.wiki.ti.com/images/7/7e/OMX_Android_GST_Comparison.pdf
>
> Regards,
> Saket Sinha