RE: proposal: use admin command (and aq) of the device to query config space

[Parav Pandit <parav@nvidia.com>]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Wednesday, August 2, 2023 9:50 AM
> 
> On Tue, Aug 01, 2023 at 07:09:14AM +0000, Parav Pandit wrote:
> > One line proposal:
> > Let's use new admin command and admin q for all device types to query
> device config space for new fields. (always).
> >
> > Details below.
> >
> > Query of device capabilities and configuration using DMA interface.
> > Need:
> > Currently device configuration space is exposed as read only registers. It is
> growing rapidly.
> > Some devices may be even multi-functionality device in coming future such as
> net + clock + rdma device.
> > For a PCI transport implementing such ever-growing capabilities,
> configuration is burdensome as plain registers.
> > Hence, it is required for the driver to query capabilities and configuration
> using a DMA interface.
> >
> > Interface requirements:
> > 1. Maintain backward compatibility for existing defined configuration fields to
> stay as registers.
> 
> This is both too specific and too vague.  I think you mean "device should be able
> to optionally support both config access over DMA and support existing
> transitional and non-transitional virtio pci drivers".
>
Not really.
New fields always over DMA, 
for example bitmap for virtio-net counters must be via dma interface and upto hash report via register access.

> > 2. Any new field added must be accessed via DMA interface, regardless of
> device implementation (hw/sw etc).
> > Results in single driver code regardless of device implementation.
> > 3. A device must be able to choose, starting from which field driver
> > must query such configuration via DMA interface. This field offset must be
> greater than currently defined configuration field.
> 
> 2 and 3 look like implementation more than like a requirement.
> Try to think what your actual requirement is.
>
It is the actual requirement to not have them as register regardless of implementation.

 
> > 4. Any driver to device query operation must not be mandated to be
> > mediated by the owner device for PCI VFs or SIOV or SF devices. Driver
> > must be able to communicate query capabilities and configuration
> > fields directly to the device regardless of device type being PCI PF, VF, SF/SIOV
> device uniformly.
> 
> I don't really know what does it mean to communicate directly with SIOV
> device.
SIOV is not any special beast. It has its own VQ and registers like VFs etc.
Communicate directly means, any mediated PF driver is not involved in communication.
PF delegate some resource as passthrough to the SIOV device and PF role ends there.
PF driver does not get involved in per command, per access transport.

> neither do I know query is and what capabilities are.
>
Capabilities = roughly config space fields.
The reason I split it because dumping everything in a single config space structure is not good as fields gets sparced.
For example, Xuan adding some 64 bits for counter and tomorrow we may need more bits.
And Heng and I will add steering bits after that.
When half year later more counter bits needed, it will be located at some different place.

If one reserved, it may be still not enough, if reserved too much its unwanted wastage.
So capabilities is basically logically clubbed structure per functionality that one can query independently of other.

Like,
struct steering_capability {

};

struct counters_bitmap {

};

 > > 5. When having multi-functionality device in future, it is desired to
> > not always query all the configuration but may be able to query per-
> functionality configurations.
> > For example, query only steering capabilities, query only rdma capabilities or
> query only clock capabilities.
> 
> I don't know what query is and what capabilities are. Do you mean "access part
> of config space"?
>
Yes, I explained above.
 
> 
> > 6. The driver should be able to query config/capabilities without
> > polling for the DMA completion, in other words, the driver should be
> > able to get notification from the device when DMA command completes.
> > 7. The driver should be able to utilize existing interrupt vector
> > and/or virtqueue for query and set operation without demanding
> > additional interrupt vector whenever possible.
> 
> additional over what? existing where?
> 
Additional over existing usage.

> >
> > There are multiple options for DMA interface.
> > Some of these options are listed below that we would like to consider fulfilling
> above requirements.
> 
> 
> Cc: Zhu Lingshan <lingshan.zhu@intel.com>
> 
> Zhu Lingshan are you still interested in a transport for SIOV (what was called
> transport vq)? Clearly SIOV needs this as part of the transport.

Its not related to SIOV.
This proposal works for PF, VF, SIOV without any difference.
And possibly for non PCI transport too when done using VQ.