Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ

[Jason Wang <jasowang@redhat.com>]

On Tue, May 9, 2023 at 11:56âAM Parav Pandit <parav@nvidia.com> wrote:
>
> > From: Jason Wang <jasowang@redhat.com>
> > Sent: Monday, May 8, 2023 11:45 PM
>
> > > The fundamental reason for not accessing the 1.x VF and SIOV device
> > > registers, config space, feature bits through PF is: it requires PF
> > > device mediation. VF and SIOV devices are first class citizen in PCIe
> > > spec and deserve direct interaction with the device.
> >
> > Unless I miss something obvious, SIOV requires mediation (or
> > composition) for sure. Otherwise you break the compatibility.
> >
> SIOV does not require mediation.
> Composition is optional like VFs.

This is not what I read from SIOV spec.

>
> > >
> > > Hence, the transport we built is to consider this in mind for the
> > > coming future.
> >
> > For transport virtqueue, it's not specific to PCI. It could be used in a much
> > broader use case.
> >
> May be.
> More below.
>
> > > So if each VF has its own configq, or cmdq, it totally make sense to
> > > me which is bootstrap interface to transport existing config space interface.
> > > The problem is: it is not backward compatible; Hence a device has no
> > > way of when to support both or only new configq.
> >
> > Providing compatibility in software is much more simpler than inventing new
> > hardware interfaces. Isn't it? (e.g if we want to provide compatibility for VF on
> > a SIOV device). And inventing a new hardware interface for compatibility might
> > not always work, it may break the advantages of the new hardware (like
> > scalability).
> >
> What I proposed below is new hardware interface for new features.
> Not for compatibility.

I'm confused, the proposal is for legacy drives so it's for
compatibility for sure. No?

> Compatibility is coming at a cost which is not scaling.
> If you attempt to scale, it breaks the future path forward to go without mediation.
>
> > >
> > > So eve growing these fields and optionally placement on configq
> > > doesn't really help and device builder to build it efficiently
> > > (without much predictability).
> >
> > Config queue is not the only choice, we have a lot of other choices (for example
> > PASID may help to reduce the on-chip resources).
> >
> PASID is for process isolation security construct, it is not for transportation.

I meant with PASID you don't even need a BAR.

>
> > >
> > > Instead of we say, that what exists today in config space stays in
> > > config space, anything additional on new q, than its deterministic
> > > behavior to size up the scale.
> >
> > Just to be clear, if we have PCI over adminq, VF's config space could be the
> > minimal one for PCI spec complaint. The real config space is accessed via the
> > admin virtqueue.
> >
> Yeah I understood what you are saying. We donât see a path forward with such mediation.
> VF and PF to have same level of direct access to the device.
>
> > >
> > > For example, a PCI device who wants to support 100 VFs, can easily
> > > size its memory to 30 bytes * 100 reserved for supporting config space.
> >
> > Those capabilities (30 bytes) can be accessed via admin virtqueue. So we don't
> > need to place them in the config space.
> >
> Same, SIOV and VFs do not prefer mediation going forward in the use cases we come across.

Unless you don't want to provide VF compatibility for SIOV.

>
> > > And new 40 bytes * 100 fields doesn't have to be in the resident memory.
> > >
> > > If we have optional configq/cmdq for transport, than 30*100 bytes are
> > > used (reserved) as 3000/(30+40) = 42 VFs.
> > >
> > > Only if some VFs use configq, more VFs can be deployed.
> >
> > I don't understand here.
> >
> Lets first clarify the non-mediated passthrough model of the VF first than come to above scale example.
>
> > > It is hard to
> > > build scale this way. Therefore suggestion is to place new attributes
> > > on new config/cmd/transport q, and old to stay as-is.
> >
> > Just to be sure we're on the same page. The proposal of both you and mine are
> > based on the adminq for PF not VF. The reason is obvious:
> > adminq per VF won't work without PASID, since it would have security issues.
> >
> The current proposal for legacy should be seen as separate and not to intermix with per VF based configuration queue.
>
> adminvq/config/control vq per VF and SIOV both devices will work without PASID (unrelated to legacy).
> Because they are like any other queue, such as txq, rxq, cvq. All of these queues are non-mediated today for PF, VF devices.
> (And so additional configq follows the natural model for config space access).

This is only true if:

1) you don't want to provide any backward compatibility for current
PCI transport, this means you need to use new drivers in the guest
2) you don't want to do live migration

If you need one of the above, PASID is a must.

Thanks