Re: [virtio] [OASIS Issue Tracker] Created: (VIRTIO-28) Implement new balloon device (ID 13)

[Rusty Russell <rusty@au1.ibm.com>]

Daniel Kiper <daniel.kiper@oracle.com> writes:
> On Tue, Oct 29, 2013 at 11:15:59PM +1030, Rusty Russell wrote:
>> Daniel Kiper <daniel.kiper@oracle.com> writes:
>> > On Tue, Oct 29, 2013 at 10:44:53AM +1030, Rusty Russell wrote:
>> >> "Michael S. Tsirkin" <mst@redhat.com> writes:
>> >> > OK there are some more issues to consider around deflate.
>> >> >
>> >> >
>> >> > 1.  On KVM, we actually want to change QEMU so that pagefaults don't
>> >> > work either.  Specifically, we want to skip pages in the balloon for
>> >> > migration.
>> >> > However, migration is done in userspace while pagefaults
>> >> > are done in kernel.
>> >> > I think the implication is that
>> >> > -	 you should be able to ask guest to inflate balloon
>> >> > 	 with pages that can be paged in (when you don't want to migrate
>> >> > 	 and want max local performance) or with pages that can not be paged in
>> >> > 	(when you want to migrate faster), dynamically, not through a
>> >> > 	device feature
>> >> > -	 "will notify before use" feature should be per a bunch or pages actually.
>> >>
>> >> I am always reluctant to implement a spec for things which don't exist.
>> >> This is the cause of the current "negative feature" mess with
>> >> VIRTIO_BALLOON_F_MUST_TELL_HOST.
>> >>
>> >> So if we *ever* want to ask for pages, let's make the the driver always
>> >> ask for pages.  You place a buffer in the queue and the device fills it
>> >> with page addresses you can now use.
>> >
>> > You mean PFNs?
>>
>> PFN << PAGE_BITS.  Since we're dealing with different size pages, using
>> exact addresses is clearer, I think.
>
> OK, it makes sens. However, I have done quick review of existing Xen balloon driver
> and PFNs or exact addresses probably will not work on Xen. Even on HVM. It must
> be confirmed. I will do that after releasing first draft.

If this is true, it means Xen cannot offer this device.  We can't have
Xen offer a device which a generic virtio-spec compliant OS can't drive.

We can, however, have a differently-numbered "Xen Balloon Device" which
takes xen-specific PFNs instead, and is otherwise identical.  This is
perfectly sane, and even the Linux driver (which will support both) will
look clean.

>> > What happen if driver request more pages than are in balloon?
>> > Are we going to support such cases? I am asking in context
>> > of memory hotplug support.
>>
>> I don't think so.  The device won't fill the entire array in that case
>> (remember, virtio gets a "used" field returned, which says how many
>> bytes were written by the device).
>>
>> Memory hotplug is properly the realm of platform specific methods
>> (eg. ACPI), so I think it's outside the virtio spec.
>
> In general ACPI is used as a notifier which says that new memory was
> installed in system. If memory hotplug is built in system it creates
> all structures needed to use newly added memory. However, memory must
> be activated by user. This process works in similar way like
> balloon deflate.
>
> Instead of ACPI another notification mechanism could be used but memory
> hotplug mechanism itself is quite generic and could be used everywhere.
> We use that features in Xen implementation. Hence, we could use the
> same in VIRTIO balloon driver. We just need device which gives pages
> to VM without any limitations (i.e. accordingly to limits established
> by admin and not limited by amount of memory assigned at boot). This
> way guest memory could be extended without stopping. If we just use
> balloon it would not work in that way.

It would have to be a feature bit: VIRTIO_BALLOON_F_UNLIMITED ?

I think it would create an extra burden for non-Xen balloon devices:
- The PFN extracted from the balloon would either be a PFN which had
   been put into the balloon, or the lowest available PFN (ie. just
   above current top of mem)

Otherwise a guest might get a crazily sparse memory map.

Thoughts?

Rusty.