Re: [virtio-dev] Re: [PATCH V2 2/2] virtio-pci: add PASID configuration extended capability

["Michael S. Tsirkin" <mst@redhat.com>]

On Mon, Jan 24, 2022 at 03:26:35PM +0000, Jean-Philippe Brucker wrote:
> On Wed, Jan 19, 2022 at 06:53:17PM -0500, Michael S. Tsirkin wrote:
> > > I think we should be explicit about this particular case because someone
> > > implementing this extension might get it wrong. MSIs should not have a
> > > PASID because IOMMUs don't support it:
> > > 
> > > * VT-d treats Requests-with-PASID to the special range 0xfeexxxxx as
> > >   normal DMA (3.14 Handling Requests to Interrupt Address Range)
> > > * AMD IOMMU reports an error for MSIs with PASID (2.2.7.7 PCIe® TLP PASID
> > >   Prefix).
> > 
> > Ouch. I didn't realize. Really weird, of the "what were they thinking"
> > variety. The natural thing would be to have PASID==VM and scope
> > both DMA and MSI within PASID. I guess that is precluded on these
> > platforms then.
> > 
> 
> I think the problem is mainly that MSIs appear as normal memory writes
> requests. If PCI had given them a special type, IOMMU/IRQ remapping
> hardware could route them more easily, but as is they only have the
> transaction's address to work with. So vendors made different choices for
> supporting MSIs with their IRQ remapping hardware. Intel and AMD define
> reserved address ranges 0xfeexxxxx, where any memory transaction is
> treated as MSI.
> 
> This is fine when the kernel is in charge of allocating the I/O address
> space for DMA, it just needs to make sure IOVAs are not allocated within
> the address region reserved for MSIs. But it's more complicated with SVA.
> 
> One of the use of PASID is assigning a partition of device (here a group
> of virtqueues) to a process. With this approach (SVA) the PASID accesses
> the process' page tables, so the virtual address space is shared between
> CPU and IOMMU, IOVA==VA. How would we deal with MSIs in this case, if they
> had a PASID?

Basically what I would do is what ARM has done ;)


>  On x86, if the IOMMU performed IRQ remapping instead of
> address translation, on a memory write with PASID to address 0xfeexxxxx,
> the process couldn't use any address in that range for DMA. The process
> would need to filter addresses returned by malloc() and treat some of them
> as non-DMA'able, which defeats the purpose of SVA. To avoid having to do
> this Intel treats all DMA with PASID as normal memory access, and the
> process can't accidentally trigger MSIs by using the wrong address. I
> think AMD behaves the same way but am not entirely sure - the wording in
> one part of the spec (2.2.7.7) seems to imply that PASID access to the MSI
> range would trigger an IOMMU error report, in which case the range would
> need carving out anyway.
> 
> > > * Arm requires creating mappings to the MSI controller in the SMMU. This
> > >   has implications for SVA where the PASID accesses the whole process
> > >   address space: if MSI transactions have a PASID prefix, that requires
> > >   mapping the MSI controller into the process address space on Arm, which
> > >   isn't convenient.
> > 
> > I'd like to understand this part a bit better. Can you explain?
> > We are talking about the IOMMU address space right?
> > What is wrong with mapping the MSI controller there?
> > Isn't convenient in what sense?
> 
> On Arm the IRQ remapping device is separate from the IOMMU, so there is no
> special address range as far as the IOMMU is concerned. An MSI goes
> through IOMMU address translation like any other memory write, the IOVA
> gets translated into a PA that corresponds to the doorbell (MMIO) address
> of the IRQ remapping device. The IOVA of the MSI is chosen by the OS and
> can have any value. Now if MSIs had a PASID, then the IOMMU would
> translate MSI addresses with the page tables corresponding to that PASID,
> which with SVA correspond to the process page tables. So the kernel would
> need to create a mapping inside the process' address space, to the PA of
> the IRQ remapping device. That's the part that is inconvenient (would not
> be a security issue because accesses to the doorbell from CPUs are ignored
> according to Arm's platform spec, but it would certainly be awkward to set
> up).
> 
> Thanks,
> Jean

The way I'd do it, is probably by an mmap call. Nothing too tricky about
that. Gives you a VA or it can get a VA with MMAP_FIXED if process
wants to allocate a range. And for a VM, we just naturally use
the regular doorbell address.

In fact, as you describe it it seems that we do have a similar problem
on AMD, we need to preclude VAs in the MSI range somehow. so I guess 2
vendors out of 3 need to carve up the process address range.

-- 
MST