Re: [virtio-dev] [PATCH v2 1/1] virtio-ism: introduce new device virtio-ism

[Xuan Zhuo <xuanzhuo@linux.alibaba.com>]

On Thu, 19 Jan 2023 13:30:09 +0100, Alexandra Winter <wintera@linux.ibm.com> wrote:
>
>
> On 16.01.23 03:10, Xuan Zhuo wrote:
> > On Fri, 13 Jan 2023 07:00:51 -0500, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >> On Fri, Jan 13, 2023 at 02:24:14PM +0800, Xuan Zhuo wrote:
> >>> On Fri, 13 Jan 2023 10:29:49 +0800, Jason Wang <jasowang@redhat.com> wrote:
> >>>> On Fri, Jan 13, 2023 at 9:59 AM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> >>>>>
> >>>>> On Thu, 12 Jan 2023 16:41:32 +0100, Halil Pasic <pasic@linux.ibm.com> wrote:
> >>>>>> On Thu, 12 Jan 2023 15:30:58 +0100
> >>>>>> Cornelia Huck <cohuck@redhat.com> wrote:
> >>>>>>
> [...]
> >>>>>>>>>
> >>>>>>>>> I'm just wondering how best to express the uniqueness scope, is it per
> >>>>>>>>> (ISM) device?
> >>>>>>>>
> >>>>>>>> No, each vm has at least one separate device. The devices in a host form
> >>>>>>>> an uniqueness scope.
> >>>>>>>
> >>>>>>> Should we call it a 'group', then? A host would be an example of such a
> >>>>>>> group.
> >>>>>>
> >>>>>> How about 'communication domain'? Devices within a single communication
> >>>>>> domain may be able to speak to each other via SMC and may not have the
> >>>>>> same device_id. Two devices from different communication domains can't
> >>>>>> communicate via ISM, but may have the same device_id.
> >>>>>
> >>>>> I agree.
> >>>>>
> >>>>>>
> >>>>>> I don't like group because it is very generic, and may sound like
> >>>>>> the grouping can be done arbitrarily. E.g. with a shared memory based
> >>>>>> implementation akin to the PoC putting devices on different hosts into
> >>>>>> the same 'group' should be illegal.
> >>>>
> >>>> Any reason why this is illegal?
> >>>
> >>> The ism device must on the same host.
> >>
> >> Fundamentally the limitation is that
> >> the devices must have access to the same memory.
> >> This is what we care about not who runs the VMs there's
> >> no need to mention that at all.
> >>
> >>
> [...]
> >>>>
> >>>>>>
> >>>>>> On the other hand there is also the following question. If we move away
> >>>>>> form the one ID per host model ("The device MUST ensure that the gid on
> >>>>>> the same entity i same and different from the gid on other entity.") then
> >>>>>> we could also allow having more than one communication domains on a
> >>>>>> single host (to limit what entities can use ISM to communicate).
> >>>>
> >>>> Yes, but I think it might not be necessary to say how gid is actually
> >>>> implemented, I can think most of the time it should be provisioned by
> >>>> the the management stack which is probably out of the scope of the
> >>>> spec.
> >>>
> >>> Imagine that the VMs from two different cloud manufacturers may have the same
> >>> GID (Host-Id). They believed that they can communicate based on ISM Device. This
> >>> is wrong.
> >>>
> >>> Thanks.
> >>
> >> Let's leave all this talk about entities out it just serves to
> >> confuse. Same as with previous discussion, explain the
> >> limitation: two devices can access the same shared memory if and
> >> only if they have the same gid. And give an example of a host
> >> running multiple VMs.
> >
> >
> [...]
> One purpose of the proposed virtio device is to work with the SMC-D(irect) protocol
> (a variant of AF_SMC sockets). I understand that this virtio device should be generic
> and usable for other purposes, but it also should be usable for SMC-D.
> SMC-D today can be used with s390's ISM devices. There we had to solve the same questions
> - How to find 2 ISM devices that can communicate?
> Let me summarize the values we use in the SMC rendevous for this purpose, maybe this
> helps this discussion.
>
> + UEID(s) â (32 byte) Defined per system (OS instance). User defined group(s) of systems that should
>      use SMC between each other.
>      If the peer has only different UEIDs: Donât try to use SMC with this peer.
>
> + SEID â (32 byte) Defined per system. Maximum space of systems that are able to use SMC-D between each other.
>    Equals to the machine hardware / first level hypervisor. Is unique per machine, derived from unique machine ID.
>     If the peer has a different SEID: Donât try to use SMC-D with this peer.
>
> + CHID â (2 Bytes) Defined per SMC-D interface. Fabric ID of this interface. Unique against other fabrics on this machine.
>     Try to find a pair of SMC-D interfaces on the same fabric for the 2 peers for this SMC-D connection.
>
> + GID â (8 bytes) Defined per SMC-D interface. ID of this interface.
>    For s390 ISM is actually globally unique. But for SMC-D protocol purpose just needs to be unique within the CHID (fabric).
>    Use it to identify the chosen interfaces to the other peer.
>
>
> You see the term GID is used for the interface, not for the fabric here.
>
> With this abstraction: Hardware, Fabric, Interface, we can say that 2 peers that are on the same
> Hardware (SEID) and can find a common Fabric (CHID) can use SMC-D to communicate and identify each
> other by Interface ID (GID).
> So for the KVM example, the fabric would be defined in the KVM host, and needs to be unique
> against other KVM hosts on the same hardware (in s390 KVM is at least the second level hipervisor)
> and against s390 ISM fabrics.

Sorry for late, most of us are on Chinese New Year vacation.

Thanks for your explanation, I think I understand.

I agree.

So the problem we have to solve now is how to unify ISM device and virtio-ism
device, which is beneficial to SMC Protocol.

For Virtio-ISM, we hope to simply return to SMC (or other applications) a ID (8 or 16 bytes)
through device, and ensure that it is unique against other hosts.

Thanks.



>
> HTH
>
>