Re: [virtio-comment] Re: [PATCH 00/11] Introduce transitional mmr pci device

["Michael S. Tsirkin" <mst@redhat.com>]

On Mon, Apr 03, 2023 at 08:25:02PM +0000, Parav Pandit wrote:
> 
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Monday, April 3, 2023 2:02 PM
> 
> > > Because vqs involve DMA operations.
> > > It is left to the device implementation to do it, but a generic wisdom
> > > is not implement such slow work in the data path engines.
> > > So such register access vqs can/may be through firmware.
> > > Hence it can involve a lot higher latency.
> > 
> > Then that wisdom is wrong? tens of microseconds is not workable even for
> > ethtool operations, you are killing boot time.
> > 
> Huh.
> What ethtool latencies have you experienced? Number?

I know an order of tens of eth calls happens during boot.
If as you said each takes tens of ms then we are talking close to a second.
That is measureable.

> > I frankly don't know, if device vendors are going to interpret "DMA" as "can
> > take insane time" then maybe we need to scrap the whole admin vq idea and
> > make it all memory mapped like Jason wanted, so as not to lead them into
> > temptation?
> 
> DMA happens for all types of devices for control and data path.
> Can you point to any existing industry specification and real implementation that highlights such timing requirements.
> This will be useful to understand where these requirements come from.
> 
> Multiple device implementors do not see memory mapped registers as way forward.
> Discussed many times.
> There is no point in going that dead end.

OK then. Then if it is a dead end then it looks weird to add a whole new
config space as memory mapped.

> > Let me try again.
> > 
> > Modern host binds to modern interface. It can use the PF normally.
> > Legacy guest IOBAR accesses to VF are translated to transport vq accesses.
> > 
> I understand this part.
> Transport VQ is on the PF, right? (Nothing but AQ, right?)
> 
> It can work in VF case with trade-off compared to memory mapped registers.
> A lightweight hypervisor cannot benefit from this which wants to utilize this for transitional PF too.
> So providing both the options is useful.

If hardware vendors do not want to bear the costs of registers then they
will not implement devices with registers, and then the whole thing will
become yet another legacy thing we need to support. If legacy emulation
without IO is useful, then can we not find a way to do it that will
survive the test of time?

> Again, I want to emphasize that register read/write over tvq has merits with trade-off.
> And so the mmr has merits with trade-off too.
> 
> Better to list them and proceed forward.
> 
> Method-1: VF's register read/write via PF based transport VQ
> Pros:
> a. Light weight registers implementation in device for new memory region window

Is that all? I mentioned more.

> Cons:
> a. Higher DMA read/write latency
> b. Device requires synchronization between non legacy memory mapped registers and legacy regs access via tvq

Same as a separate mmemory bar really.  Just don't do it. Either access
legacy or non legacy.

> c. Can only work with the VF. Cannot work for thin hypervisor, which can map transitional PF to bare metal OS
> (also listed in cover letter)

Is that a significant limitation? Why?

> Method-2: VF's register read/write via MMR (current proposal)
> Pros:
> a. Device utilizes the same legacy and non-legacy registers.

Not really. For starters endian-ness can be different. Maybe for some
devices and systems. 

> b. an order of magnitude lower latency due to avoidance of DMA on register accesses
> (Important but not critical)

And no cons? Even if you could not see them yourself did I fail to express myself to such
an extent?

> > > No. Interrupt latency is in usec range.
> > > The major latency contributors in msec range can arise from the device side.
> > 
> > So you are saying there are devices out there already with this MMR hack
> > baked in, and in hardware not firmware, so it works reasonably?
> It is better to not assert a solution a "hack",

Sorry if that sounded offensive.  a hack is not necessary a bad thing.
It's a quick solution to a very local problem, though.

> when you are still
> trying to understand the trade-offs of multiple solutions and when you
> are yet to fully review all requirements.
> (and when solution is also based on an offline feedback from you!)

Sorry if I set you on a wrong path I frankly didn't realize how
big a change the result will be. I was thinking more along the lines
of a capability describing a portion of a memory bar, saying access
there is equivalent to access to the io bar. That's it.

> No. I didn't say that device is out there.
> However, large part of the proposed changes is done based on real devices (and not limited to virtio).

Yes motivation is one of the things I'm trying to work out here.
It does however not help that it's an 11 patch strong patchset
adding 500 lines of text for what is supposedly a small change.

> Regarding tvq, I have some idea on how to improve the register read/writes so that its optimal for devices to implement.

Sounds useful, and maybe if tvq addresses legacy need then focus on
that?

-- 
MST