RE: [PATCH v2] Add device reset timeout field

[Parav Pandit <parav@nvidia.com>]

> From: Cornelia Huck <cohuck@redhat.com>
> Sent: Monday, October 11, 2021 9:52 PM
> 
> On Mon, Oct 11 2021, Parav Pandit <parav@nvidia.com> wrote:
> 
> >> From: Cornelia Huck <cohuck@redhat.com>
> >> Sent: Monday, October 11, 2021 8:29 PM
> >>
> >> On Mon, Oct 11 2021, Parav Pandit <parav@nvidia.com> wrote:
> 
> >> > I think read only device reset timeout is most elegant option
> >> > during device
> >> initialization phase that eliminates infinite loop of today.
> >>
> >> Why can't a driver just go ahead and do a timeout regardless?
> > o.k. lets consider this thought exercise. What is the timeout value that driver
> will choose if device doesn't specify one?
> > I explained in previous thread and you acked that actual fw based device
> may take longer to initialize than pure sw implementation backend.
> > In second example a pre-boot device can take even longer initialization time.
> > Sriov VF device may initialize lot faster.
> > Instead of driver having such transport, and device specific checks, (or some
> very short or very long timeout), we propose, that let device mention such
> timeout value.
> 
> But does the driver really care in all cases? 
Idea is that driver shouldn't care based on device type; rather just care what device provided.

> Say, I'm a virtio-blk driver, and I
> drive all kinds of virtio-blk devices. If I choose a timeout value that
> accommodates hardware devices, software devices will just always breeze
> past, and nothing bad happens. 
I had hard time defining a universal timeout that fits all device types of hw and sw.
We have 3 variants of the hw based devices and they have range from milliseconds to ten of seconds (in worst case).
Hence, the request is to let device feed the value based on the internal logic.

> If I know that I'm in a pre-boot environment,
> I'll just add on a value that allows devices to warm up. I'd say reasonable
> values depend way more on the driver implementation and the environment
> it is running in than on the device.
> 
It will be usually min_of (env.max_timeout, device.reset_timeout) if env wants to enforce its upper limit.

> >
> >> This seems
> >> pretty much to depend on the implementation. Nothing in the virtio
> >> spec forces the driver to reset all devices serially and wait
> >> potentially forever. (If you deal with a large amount of subdevices
> >> you cannot access concurrently, you likely already have problems.)
> > Sure, nothing in driver forces serial initialization.
> > However in reality, PCI device scanning on the bus in hotplug driver, preboot
> env, sriov vf initialization is serial.
> 
> So, can't pci scan asynchronously? Is that a limitation in the pci spec?
There isn't limitation from pci spec to my knowledge.
Linux device discovery to my knowledge is not parallel across or within a subsystem.

> The Linux ccw bus switched to asynchronous processing precisely because of
> slow devices years ago, FWIW, so that may be why I don't really understand
> the issue.
> 
> > In preboot case, even the order is defined, so it is sort of serial.
> 
> Again, is that somewhere in the hardware spec, and can it be fixed?
It is not in the hardware spec.
Preboot env usually runs with limited text and data memory without the OS.
User defined the boot order at the BIOS level. It is the way some systems work.
virtio device is fitting in the existing eco-system of physical servers.

> Serializing something that might take some time just seems troublesome.
> 
True, which is why as Michael mentioned, device resetting within short interval is better.
I cannot promise, but yes, with newer generation usually such timing and functionality should improves.
There is still finite timeout though.

> > In other uses, concurrent access is fine and host does that today when it
> disables autoprobe on the PCI Bus and probes them in parallel after enabling
> sriov.
> > I hope you understand that there are both use-cases exists which scan
> parallel and some serial.
> 
> Can at least some processing be made asynchronous? Like, you discover the
> device, but then do the virtio setup out of band.
Do you mean differ virtio device setup in a workqueue context or similar?

This may be possible, but when user does sysfs operations, it adds uncertainly to when the device's netdevice will actually appear even though sysfs operation completed for probe.
It requires users to listen to udev events to know that for a given PCI virtio pci device, now a net/block device is available.
This makes the whole flow complex to use.
May be certain device omit exposing device reset timeout, and workqueue/differed context can be avoided, but again now the user sw is unaware when to use udev to discover vs inline.