Re: [virtio] [OASIS Issue Tracker] Created: (VIRTIO-28) Implement new balloon device (ID 13)

[Daniel Kiper <daniel.kiper@oracle.com>]

On Thu, Dec 12, 2013 at 11:40:11AM +1030, Rusty Russell wrote:
> Daniel Kiper <daniel.kiper@oracle.com> writes:
> > On Tue, Oct 29, 2013 at 11:15:59PM +1030, Rusty Russell wrote:
> >> Daniel Kiper <daniel.kiper@oracle.com> writes:
> >> > On Tue, Oct 29, 2013 at 10:44:53AM +1030, Rusty Russell wrote:
> >> >> "Michael S. Tsirkin" <mst@redhat.com> writes:
> >> >> > OK there are some more issues to consider around deflate.
> >> >> >
> >> >> >
> >> >> > 1.  On KVM, we actually want to change QEMU so that pagefaults don't
> >> >> > work either.  Specifically, we want to skip pages in the balloon for
> >> >> > migration.
> >> >> > However, migration is done in userspace while pagefaults
> >> >> > are done in kernel.
> >> >> > I think the implication is that
> >> >> > -	 you should be able to ask guest to inflate balloon
> >> >> > 	 with pages that can be paged in (when you don't want to migrate
> >> >> > 	 and want max local performance) or with pages that can not be paged in
> >> >> > 	(when you want to migrate faster), dynamically, not through a
> >> >> > 	device feature
> >> >> > -	 "will notify before use" feature should be per a bunch or pages actually.
> >> >>
> >> >> I am always reluctant to implement a spec for things which don't exist.
> >> >> This is the cause of the current "negative feature" mess with
> >> >> VIRTIO_BALLOON_F_MUST_TELL_HOST.
> >> >>
> >> >> So if we *ever* want to ask for pages, let's make the the driver always
> >> >> ask for pages.  You place a buffer in the queue and the device fills it
> >> >> with page addresses you can now use.
> >> >
> >> > You mean PFNs?
> >>
> >> PFN << PAGE_BITS.  Since we're dealing with different size pages, using
> >> exact addresses is clearer, I think.
> >
> > OK, it makes sens. However, I have done quick review of existing Xen balloon driver
> > and PFNs or exact addresses probably will not work on Xen. Even on HVM. It must
> > be confirmed. I will do that after releasing first draft.
>
> If this is true, it means Xen cannot offer this device.  We can't have
> Xen offer a device which a generic virtio-spec compliant OS can't drive.
>
> We can, however, have a differently-numbered "Xen Balloon Device" which
> takes xen-specific PFNs instead, and is otherwise identical.  This is
> perfectly sane, and even the Linux driver (which will support both) will
> look clean.

This make sens but I think that it is last resort solution. IMO all
devices/drivers should work on any hypervisor. I will play with new
VIRTIO balloon device/driver and we will see what is possible.

> >> > What happen if driver request more pages than are in balloon?
> >> > Are we going to support such cases? I am asking in context
> >> > of memory hotplug support.
> >>
> >> I don't think so.  The device won't fill the entire array in that case
> >> (remember, virtio gets a "used" field returned, which says how many
> >> bytes were written by the device).
> >>
> >> Memory hotplug is properly the realm of platform specific methods
> >> (eg. ACPI), so I think it's outside the virtio spec.
> >
> > In general ACPI is used as a notifier which says that new memory was
> > installed in system. If memory hotplug is built in system it creates
> > all structures needed to use newly added memory. However, memory must
> > be activated by user. This process works in similar way like
> > balloon deflate.
> >
> > Instead of ACPI another notification mechanism could be used but memory
> > hotplug mechanism itself is quite generic and could be used everywhere.
> > We use that features in Xen implementation. Hence, we could use the
> > same in VIRTIO balloon driver. We just need device which gives pages
> > to VM without any limitations (i.e. accordingly to limits established
> > by admin and not limited by amount of memory assigned at boot). This
> > way guest memory could be extended without stopping. If we just use
> > balloon it would not work in that way.
>
> It would have to be a feature bit: VIRTIO_BALLOON_F_UNLIMITED ?

OK.

> I think it would create an extra burden for non-Xen balloon devices:
> - The PFN extracted from the balloon would either be a PFN which had
>    been put into the balloon, or the lowest available PFN (ie. just
>    above current top of mem)
>
> Otherwise a guest might get a crazily sparse memory map.
>
> Thoughts?

I used this simplistic solution in Xen balloon driver and it does not
work well if there is an I/O region above max_pfn. So we should not
make this mistake here. I think that we could use solution similar
to used on real hardware with ACPI. In memory hotplug case VIRTIO
balloon device should send start address of hotplugged region and its
size. Later everything should work as in pure balloon driver.

Daniel