RE: [virtio-comment] Re: [PATCH 09/11] transport-pci: Describe PCI MMR dev config registers

[Parav Pandit <parav@nvidia.com>]

> From: Jason Wang <jasowang@redhat.com>
> Sent: Sunday, April 16, 2023 11:23 PM
> 
> On Fri, Apr 14, 2023 at 11:51âAM Parav Pandit <parav@nvidia.com> wrote:
> >
> >
> >
> > > From: Jason Wang <jasowang@redhat.com>
> > > Sent: Thursday, April 13, 2023 11:38 PM
> >
> > > > > 1) spec doesn't enforce the size of a specific structure
> > > > Spec will be extended in coming time.
> > >
> > > It's too late to do any new restriction without introducing a flag.
> > We are really diverging from the topic.
> > I donât think it is late. The work in this area of PCI VF has not even begun fully.
> 
> I meant it needs a new feature flag.
> 
Ok.
> >
> > > Mandating size may easily end up with a architecte specific solution.
> > >
> > Unlikely. Other standard device types are also expanding this way.
> 
> I think we are talking about software technologies instead of device design
> here.
> 
Isn't the size of BAR and its cap_len exposed by the device?

> For devices it works.But for hypervisor, it needs to deal with the size that
> doesn't match arch's page size.
> 
PCI BAR size of the VF can learn the system page size being different for x86 (4K) and arm (64K).
PCI transport seems to support it.

PCI PF on bare-metal has to understand the highest page size anyway if for some reason bare-metal host wants to map this PF to the VM.

A hypevisor mediating and emulating needs to learn the system page size anyway.
If underlying device page size is smaller, hypevisor may end up mediating it.

> I meant you can't have recommendations in features and config. 
Sure. There is none.

> What's more,
> assuming you have two generations of device
> 
> gen1: features x,y
> gen2: features x,y,z
> 
> You won't be able to do migration between gen1 and gen2 without mediation.
Gen1 can easily migrate to gen2, because gen1 has smaller subset than gen2.
When gen2 device is composed, feature z is disabled.

Gen2 to gen1 migration can do software-based migration anyway or through mediation.
But because gen2 may need to migrate to gen1, hence gen2 to gen2 migration also should be done through mediation, doesnât make sense to me.

> Such technologies have been used by cpu features for years.
> I am not sure why it became a problem for you.
> 
> > Apart from it some of the PCI device layout compat checks will be covered
> too.
> >
> > > And what you proposed is to allow the management to know the exact
> > > hardware layout in order to check the compatibility? And the
> > > management needs to evolve as new structures are added.
> > Mostly not mgmt stack may not need to evolve a lot.
> > Because most layouts should be growing within the device context and not at
> the PCI capabilities etc area.
> >
> > And even if it does, its fine as large part of it standard PCI spec definitions.
> 
> So as mentioned in another thread, this is a PCI specific solution:
> 
> 1) feature and config are basic virtio facility
> 2) capability is not but specific to PCI transport
> 
So any LM solution will have transport specific checks and virtio level checks.

> Checking PCI capability layout in the virtio management is a layer violation
> which can't work for future transport like SIOV or adminq.
Virtio management that will have transport level checks is not a violation.
SIOV will define its own transport anyway. Not to mix with ccw/mmio or pci.

> Management should only see virtio device otherwise the solution becomes
> transport specific.
> 
Solution needs to cover transport as well as transport is integral part of the virtio spec.
Each transport layer will implement feature/config/cap in its own way.

> >
> > > This complicates the work of
> > > management's furtherly which I'm not sure it can work.
> > >
> > Well, once we work towards it, it can work. :)
> >
> > > >
> > > > > Hypervisor needs to start from a mediation method and do BAR
> > > > > assignment only when possible.
> > > > >
> > > > Not necessarily.
> > > >
> > > > > > Cons:
> > > > > > a. More AQ commands work in sw
> > > > >
> > > > > Note that this needs to be done on top of the transport virtqueue.
> > > > > And we need to carefully design the command sets since they
> > > > > could be mutually exclusive.
> > > > >
> > > > Not sure what more to expect out of transport virtqueue compared to AQ.
> > > > I didnât follow, which part could be mutually exclusive?
> > >
> > > Transport VQ allows a modern device to be transported via adminq.
> > >
> > May be for devices it can work. Hypervisor mediation with CC on horizon for
> new capabilities is being reduced.
> > So we donât see transport vq may not be path forward.
> >
> > > And you want to add commands to transport for legacy devices.
> > >
> > Yes only legacy emulation who do not care about hypervisor mediation.
> >
> > > Can a driver use both the modern transport commands as well as the
> > > legacy transport commands?
> > Hard to answer, I likely do not understand as driver namespace is unclear.
> 
> Things might be simplified if we use separate queues for admin, transport and
> legacy.
> 
Do you mean say we have three AQs, AQ_1, AQ_2, and AQ_3;
AQ_1 of the PF used by admin work as SIOV device create, SRIOV MSIX configuration.
AQ_2 of the PF used for transporting legacy config access of the PCI VF
AQ_3 of the PF for some transport work.

If yes, sounds find to me.