Re: [virtio-comment] Re: [PATCH] virtio-balloon: add an untrusted device feature

[Keir Fraser <keirf@google.com>]

On Wed, Aug 03, 2022 at 07:31:06AM -0400, Michael S. Tsirkin wrote:
> On Wed, Aug 03, 2022 at 08:44:41AM +0000, Keir Fraser wrote:
> > This flag indicates to the
> > driver that memory ownership must be relinquished via the TCB, by
> > platform-specific means, before reporting that memory to the device.
> 
> Thanks for the proposal! I feel it needs to be more detailed to be
> useful. In particular we need a bit more info about TCB and memory
> ownership here I feel, so people can interpret this. Is there a spec
> link?

The specific motivation for this is PKVM on ARM, where protected VM
memory is inaccessible to the host kernel. Ownership is tracked by a
small hypervisor, and that hypervisor needs to be aware that the pVM
is willingly returning memory to the host. There's some old
description of ARM PKVM here https://lwn.net/Articles/836693/

So the purpose of the new flag is:

1. To inform the pVM that it must notify the hypervisor (although
there could be out-of-band mechanisms to detect/probe this)

2. If the flag is not accepted by the driver, device can abort balloon
setup (since pVM ballooning cannot work without hypervisor
notifications).

> > +\item[ VIRTIO_BALLOON_F_UNTRUSTED_DEVICE(6) ] The driver must
> > +    relinquish memory ownership via the Trusted Computing Base (TCB) before
> > +    notifying the device.
> 
> The name does not seem to match what it does.

Maybe something more direct, like VIRTIO_BALLOON_F_NOTIFY_HYP?

> > +If guest memory is protected from the host,
> 
> This does not make it clear how this is different from VIRTIO_F_ACCESS_PLATFORM
> which says:
> 
> \item[VIRTIO_F_ACCESS_PLATFORM(33)] This feature indicates that
>   the device can be used on a platform where device access to data
>   in memory is limited and/or translated.

Using this existing flag could possibly work, assuming there are no
conflicting legacy semantics for this flag in relation to balloon
driver.  At the moment the Linux virtio-balloon driver unconditionally
clears this feature flag, so perhaps there are no legacy semantics at
all.

We could instead accept this existing flag and use it to turn on the
hypervisor notifications as well as using DMA API for vring setup (we
do need both anyway)? If the flag is cleared, as with the existing
kernel driver, the device knows for sure that the driver is not ARM
PKVM aware.

"Turn on hypervisor notifications" here is platform specific: In ARM
PKVM it means probe for existence of the new notification hypercall
and send notifications if the hypercall exists.

I'd be more than happy to use this flag rather than introduce
another. Multiplying these sorts of flags more than absolutely
necessary is for sure not a good idea.

> > @@ -5647,6 +5664,10 @@ \subsection{Device Operation}\label{sec:Device Types / Memory Balloon Device / D
> >      pages. These addresses are divided by 4096\footnote{This is historical, and independent of the guest page size.
> >  } and the descriptor
> >      describing the resulting 32-bit array is added to the inflateq.
> > +  \item If the VIRTIO_BALLOON_F_UNTRUSTED_DEVICE feature has been
> > +  negotiated, the driver MUST relinquish memory ownership via the TCB
> > +  before adding it to the inflateq.
> > +
> >    \end{enumerate}
> >  
> >  \item To remove memory from the balloon (aka. deflate):
> 
> Don't we need to take it back before deflate?

No, giving memory to a protected VM does not need the VM's explicit permission.

If such permission were required then we would have a problem implementing free page reporting (and maybe hinting too).

> > @@ -6105,6 +6126,9 @@ \subsubsection{Free Page Reporting}\label{sec:Device Types / Memory Balloon Devi
> >  driver MUST initialize all free pages with \field{poison_val} before
> >  reporting them.
> >  
> > +If the VIRTIO_BALLOON_F_UNTRUSTED_DEVICE feature has been negotiated, the
> > +driver MUST notify free pages to the TCB before reporting them.
> > +
> >  The driver MUST NOT use the reported pages until the device has
> >  acknowledged the reporting request.
> 
> Should this affect hinting too?

Yes you are probbaly right. Though given the main usage of hinting is
for live migration, it's unlikely to be enabled for ARM PKVM any time
soon. I don't know how we'd live migrate a VM with confidential memory.

> Is "notify" same as "relinquish ownership"?

Yes, sorry, I should tighten up the wording.

> Are these terms defined in some spec?

A spec for relinquishing memory ownership in ARM PKVM? I could
certainly put something together if it's helpful. Also, concretely,
there are patches to the Linux kernel.

I assumed it would make sense to address the virtio spec before
floating implementation patches, but perhaps it makes sense to
consider it all in the round?

Also, if we can agree that VIRTIO_F_ACCESS_PLATFORM can be pressed
into service for our needs, perhaps the spec changes become a bit more
straightforward. Though I assume you would still want many of the
above additions, to make it clear how a protected VM should interact
with ballooning?

> >  
> > -- 
> > 2.37.1.455.g008518b4e5-goog
> 
> 
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
> 
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
> 
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>