Re: [virtio] New virtio balloon...

[Rusty Russell <rusty@au1.ibm.com>]

Daniel Kiper <daniel.kiper@oracle.com> writes:
> On Thu, Jan 30, 2014 at 07:34:30PM +1030, Rusty Russell wrote:
>> Hi,
>>
>>         I tried to write a new balloon driver; it's completely untested
>> (as I need to write the device).  The protocol is basically two vqs, one
>> for the guest to send commands, one for the host to send commands.
>>
>> Some interesting things come out:
>> 1) We do need to explicitly tell the host where the page is we want.
>>    This is required for compaction, for example.
>>
>> 2) We need to be able to exceed the balloon target, especially for page
>>    migration.  Thus there's no mechanism for the device to refuse to
>>    give us the pages.
>
> Admin should have a way to impose a mem limit on guest. However,
> he/she should be able to change it in any direction (up and down) and
> even increase it above limit established at guest boot (needed for
> memory hotplug). On the other hand guest should not be able to allocate
> more memory then it was requested by admin in a given time.

Well, now we have VIRTIO_BALLOON_GCMD_EXCHANGE_PAGES, one problem with a
strict limit is gone.  We still have the problem of a race of the host
lowering the target while the guest makes a request for more pages, but
perhaps we just allow a single such request?

>> 3) The device can offer multiple page sizes, but the driver can only
>>    accept one.  I'm not sure if this is useful, as guests are either
>>    huge page backed or not, and returning sub-pages isn't useful.
>
> Hmmm... I suppose that even if guest is backed by huge pages then internaly
> it uses standard page sizes (if not directed otherwise). So we have a
> problem here because I do not know what to do if guest backed by 1 GiB
> pages would like to inflate balloon with 4 KiB pages. Should we refuse
> that?

Two choices: offer 1G pages to the guest.  If it can't handle that, it's
pretty useless anyway (and will fail initialization).  Otherwise, offer
both 1G and 4k pages, and it might accept 4k pages (you'd do this if you
have the ability to split 1G pages into 4k pages, I guess).

>> +/* define the balloon_mapping->a_ops callback to allow balloon page migration */
>> +static const struct address_space_operations virtio_balloon_aops = {
>> +			.migratepage = virtballoon_migratepage,
>> +};
>> +#endif /* CONFIG_BALLOON_COMPACTION */
>
> Do we really need this feature on guest?

Well, it's really a Linux-specific thing, but yes, if you can't migrate
pages then page compation really suffers.  Rafael Aquini
<aquini@redhat.com> added this.

>> +/* This means the balloon can go negative (ie. add memory to system) */
>> +#define VIRTIO_BALLOON_F_EXTRA_MEM	0
>> +
>> +struct virtio_balloon_config_space {
>> +	/* Set by device: bits indicate what page sizes supported. */
>> +	__le64 pagesizes;
>> +	/* Set by driver: only a single bit is set! */
>> +	__le64 page_size;
>> +
>> +	/* These set by device if VIRTIO_BALLOON_F_EXTRA_MEM. */
>> +	__le64 extra_mem_start;
>> +	__le64 extra_mem_end;
>
> This cannot be a part of config space. Guest should be able to hotplug
> memory many times. Hence it should be a part of reply from host.

This was to specify the upper limits of where the extra mem is.  It was
intended to represent one or more section sizes.

> Additionally,
> we should remember that memory is hotplugged in chunks known as section size.
> They are usually quite big and architecture depended (e.g. IIRC it is 128 MiB
> on x86_64). So maybe guest should tell host about its supported section size
> in config space. However, there should not be a requirement that target must
> be equal to multiple of section size in case of memory hotplug. It should be
> set as needed and balloon driver should reserve relevant memory region told
> by host (it should be rounded by host up to nearest multiple of section size)
> and later back relevant pages with PFNs up to a given target.

We could either have a special request (VIRTIO_BALLOON_GCMD_NEW_MEM)
where the guest specifies where it wants another chunk of memory.  Then
after that, it can ask for those pages' PFNs in
VIRTIO_BALLOON_GCMD_GET_PAGES.

Or we could simply allow a guest to request (if the
VIRTIO_BALLOON_F_EXTRA_MEM feature is negotiated) any PFN it wants, and
let it handle its sections itself.

The latter is simpler, but is it sufficient?

Cheers,
Rusty.