Re: [virtio-comment] Re: [PATCH 09/11] transport-pci: Describe PCI MMR dev config registers

[Jason Wang <jasowang@redhat.com>]

On Fri, Apr 14, 2023 at 11:51âAM Parav Pandit <parav@nvidia.com> wrote:
>
>
>
> > From: Jason Wang <jasowang@redhat.com>
> > Sent: Thursday, April 13, 2023 11:38 PM
>
> > > > 1) spec doesn't enforce the size of a specific structure
> > > Spec will be extended in coming time.
> >
> > It's too late to do any new restriction without introducing a flag.
> We are really diverging from the topic.
> I donât think it is late. The work in this area of PCI VF has not even begun fully.

I meant it needs a new feature flag.

>
> > Mandating size may easily end up with a architecte specific solution.
> >
> Unlikely. Other standard device types are also expanding this way.

I think we are talking about software technologies instead of device
design here.

For devices it works.But for hypervisor, it needs to deal with the
size that doesn't match arch's page size.

>
> > >
> > > > 2) bing vendor locked thus a blocker for live migration as it
> > > > mandate the layout for the guest, mediation layer is a must in this
> > > > case to maintain cross vendor compatibility
> > > >
> > > With PCI VF will have compatibility check and also vendor recommendations.
> >
> > I'm not sure what type of recommendations you want.
> >
> > We've already had:
> >
> > 1) features
> > 2) config
> >
> Those cover the large part.

I meant you can't have recommendations in features and config. What's
more, assuming you have two generations of device

gen1: features x,y
gen2: features x,y,z

You won't be able to do migration between gen1 and gen2 without
mediation. Such technologies have been used by cpu features for years.
I am not sure why it became a problem for you.

> Apart from it some of the PCI device layout compat checks will be covered too.
>
> > And what you proposed is to allow the management to know the exact
> > hardware layout in order to check the compatibility? And the management
> > needs to evolve as new structures are added.
> Mostly not mgmt stack may not need to evolve a lot.
> Because most layouts should be growing within the device context and not at the PCI capabilities etc area.
>
> And even if it does, its fine as large part of it standard PCI spec definitions.

So as mentioned in another thread, this is a PCI specific solution:

1) feature and config are basic virtio facility
2) capability is not but specific to PCI transport

Checking PCI capability layout in the virtio management is a layer
violation which can't work for future transport like SIOV or adminq.
Management should only see virtio device otherwise the solution
becomes transport specific.

>
> > This complicates the work of
> > management's furtherly which I'm not sure it can work.
> >
> Well, once we work towards it, it can work. :)
>
> > >
> > > > Hypervisor needs to start from a mediation method and do BAR
> > > > assignment only when possible.
> > > >
> > > Not necessarily.
> > >
> > > > > Cons:
> > > > > a. More AQ commands work in sw
> > > >
> > > > Note that this needs to be done on top of the transport virtqueue.
> > > > And we need to carefully design the command sets since they could be
> > > > mutually exclusive.
> > > >
> > > Not sure what more to expect out of transport virtqueue compared to AQ.
> > > I didnât follow, which part could be mutually exclusive?
> >
> > Transport VQ allows a modern device to be transported via adminq.
> >
> May be for devices it can work. Hypervisor mediation with CC on horizon for new capabilities is being reduced.
> So we donât see transport vq may not be path forward.
>
> > And you want to add commands to transport for legacy devices.
> >
> Yes only legacy emulation who do not care about hypervisor mediation.
>
> > Can a driver use both the modern transport commands as well as the legacy
> > transport commands?
> Hard to answer, I likely do not understand as driver namespace is unclear.

Things might be simplified if we use separate queues for admin,
transport and legacy.

> Let me try.
>
> A modern driver in guest VM accessing a transitional device has its own transport vq for say config space RW.
> This transport queue can directly reach to the device without hypervisor mediation, - yes.
> Same device if accessed via some legacy guest VM driver, it will gets its config space accessed via transport VQ (AVQ) of the hypervisor PF AQ.

Not only the config space, but we can revisit those issues when we
agree to use adminq (or other like PASID which is even more flexible).

Thanks