Re: [virtio-comment] [PATCH v2 0/8] Introduce device migration support commands

[Jason Wang <jasowang@redhat.com>]

On Wed, Oct 18, 2023 at 4:07âAM Parav Pandit <parav@nvidia.com> wrote:
>
> This series introduces administration commands for member device migration
> for PCI transport; when needed it can be extended for other transports
> too.
>
> Use case requirements:
> ======================
> 1. A hypervisor system needs to provide a PCI VF as passthrough
>    device to the guest virtual machine and also support live
>    migration of this virtual machine.
>    A passthrough device has typically only PCI configuration space
>    and MSI-X table emulated. No virtio native interface offered
>    by the virtio member device is trapped and/or emulated.
> 2. A virtual machine may have one or more such passthrough
>    virtio devices.
> 3. A virtual machine may have other PCI passthrough device
>    which may also interact with virtio device.
> 4. A hypervisor runs a generic device type agnostic driver with
>    extension to support device migration.
> 5. A PCI VF passthrough device needs to support transparent
>    device reset and PCI FLR while the device migration is
>    ongoing.
> 6. A owner driver do not involve in device operations mediation
>    for the passthrough device at virtio interface level.
> 7. Mechanism is generic enough that applies to large family of
>    virtio devices and it does not involve trapping any virtio
>    device interfaces for the passthrough devices.
>
> Overview:
> =========
> Above usecase requirements is solved by PCI PF group owner driver
> facilitating the member device migration functionality using
> administration commands.
>
> There are three major functionalities.
>
> 1. Suspend and resume the device operation
> 2. Read and Write the device context containing all the information
>    that can be transferred from source to destination to migrate to
>    a member device
> 3. Track pages written by the device during device migration is
>    ongoing
>
> This comprehensive series introduces 4 infrastructure pieces
> covering PCI transport, peer to peer PCI devices, page write tracking
> (aka dirty page tracking) and generic virtio device context.
>
> 1. Device mode get,set (active, stop, freeze)
> 2. Device context read and write
> 3. Defines device context and compatibility command
> 4. Write reporting to track page addresses
>
> This series enables virtio PCI SR-IOV member device to member device
> migration. It can also be used to/from migrate from PCI SR-IOV member
> device to software composed PCI device if/when needed which can
> parse and compose software based PCI virtio device.
>
> Example flow:
> =============
> Source hypervisor:
> 1. Instructs device to start tracking pages it is writing
> 2. Periodically query the addresses of the written pages
> 3. Suspend the device operation
> 4. Read the device context and transfer to destination hypervisor
>
> Destination hypervisor:
> 5. Write the device context received from source
> 6. Resume the device that has newly written device context
>
> Patch summary:
> ==============
> patch-1: Adds theory of operation for device migration commands
> patch-2: Redefine reserved2 to command output field
> patch-3: Defines short device context for split virtqueues
> patch-4: Adds device migration commands
> patch-5: Adds requirements for device migration commands
> patch-6: Adds theory of operation for write reporting commands
> patch-7: Adds write reporting commands
> patch-8: Adds requirements for write reporting commands
>
> It also takes inspiration from the similar idea presented at KVM Forum
> at [1].
>
> Please review.
>
> Changelog:
> ==========
> v1->v2:
> - Addressed comments from Michael and Jason
> - replaced iova to page/physical address range in write recording commands
> - several device specific requirements added to clarify, interaction of
>   device reset, FLR, PCI PM and admin commands
> - added device context fields query command to learn compatibility
> - split device context field type range into generic and device specific
> - added device context extension section to maintain backward and future
>   compatibility
> - several rewording in theory of operation
> - added requirements to cover config space read/write interaction with
>   device context commands
> - added assumption about pci config space and msix table not present in
>   device context, which can be added when hypervisor need them
> v0->v1:
> - enrich device context to cover device configuration layout, feature bits
> - fixed alignment of device context fields
> - added missing Sign-off for the joint work done with Satananda
> - added link to the github issue

A lot of questions were not answered in V1, do you expect me to repeat
it again here?

Thanks