Re: [virtio] [OASIS Issue Tracker] Created: (VIRTIO-28) Implement new balloon device (ID 13)

[Daniel Kiper <daniel.kiper@oracle.com>]

On Tue, Oct 29, 2013 at 11:15:59PM +1030, Rusty Russell wrote:
> Daniel Kiper <daniel.kiper@oracle.com> writes:
> > On Tue, Oct 29, 2013 at 10:44:53AM +1030, Rusty Russell wrote:
> >> "Michael S. Tsirkin" <mst@redhat.com> writes:
> >> > OK there are some more issues to consider around deflate.
> >> >
> >> >
> >> > 1.  On KVM, we actually want to change QEMU so that pagefaults don't
> >> > work either.  Specifically, we want to skip pages in the balloon for
> >> > migration.
> >> > However, migration is done in userspace while pagefaults
> >> > are done in kernel.
> >> > I think the implication is that
> >> > -	 you should be able to ask guest to inflate balloon
> >> > 	 with pages that can be paged in (when you don't want to migrate
> >> > 	 and want max local performance) or with pages that can not be paged in
> >> > 	(when you want to migrate faster), dynamically, not through a
> >> > 	device feature
> >> > -	 "will notify before use" feature should be per a bunch or pages actually.
> >>
> >> I am always reluctant to implement a spec for things which don't exist.
> >> This is the cause of the current "negative feature" mess with
> >> VIRTIO_BALLOON_F_MUST_TELL_HOST.
> >>
> >> So if we *ever* want to ask for pages, let's make the the driver always
> >> ask for pages.  You place a buffer in the queue and the device fills it
> >> with page addresses you can now use.
> >
> > You mean PFNs?
>
> PFN << PAGE_BITS.  Since we're dealing with different size pages, using
> exact addresses is clearer, I think.

OK, it makes sens. However, I have done quick review of existing Xen balloon driver
and PFNs or exact addresses probably will not work on Xen. Even on HVM. It must
be confirmed. I will do that after releasing first draft.

> >> +3. To withdraw pages from the balloon, the same structure should be
> >> +   placed in the todevq queue, with the page array writable:
> >> +
> >> +	struct virtio_balloon_pages {
> >> +#define VIRTIO_BALLOON_REQ_PAGES	2
> >> +		u32 type; // VIRTIO_BALLOON_REQ_PAGES
> >> +		u64 page[];
> >
> > What is the size of this array?
>
> It's implied by the length of the request.

OK.

> >> +	};
> >> +
> >> +   The device may not fill the entire page array.  The contents
> >> +   of the pages received will be undefined.  The device should
> >>     keep count of how many pages remain in the balloon so it can
> >>     correctly respond to future resize requests.
> >
> > What happen if driver request more pages than are in balloon?
> > Are we going to support such cases? I am asking in context
> > of memory hotplug support.
>
> I don't think so.  The device won't fill the entire array in that case
> (remember, virtio gets a "used" field returned, which says how many
> bytes were written by the device).
>
> Memory hotplug is properly the realm of platform specific methods
> (eg. ACPI), so I think it's outside the virtio spec.

In general ACPI is used as a notifier which says that new memory was
installed in system. If memory hotplug is built in system it creates
all structures needed to use newly added memory. However, memory must
be activated by user. This process works in similar way like
balloon deflate.

Instead of ACPI another notification mechanism could be used but memory
hotplug mechanism itself is quite generic and could be used everywhere.
We use that features in Xen implementation. Hence, we could use the
same in VIRTIO balloon driver. We just need device which gives pages
to VM without any limitations (i.e. accordingly to limits established
by admin and not limited by amount of memory assigned at boot). This
way guest memory could be extended without stopping. If we just use
balloon it would not work in that way.

> Ballooning is simpler, but has shown to be useful.  Both can coexist.

Right. We did that in Xen balloon driver. It supports ballooning but
memory hotplug is used when more memory is requested than it was
assigned to VM at boot. Later all stuff works as usual. Even balloon
driver works in the same way on hotplugged memory like on memory
allocated at boot. So we just need a device which gives pages
to VM without any limitations.

Of course we could consider another device for memory hotplug but
I do not think it makes sens.

Daniel