Re: [virtio-dev] Re: [virtio-comment] [PATCH v3 1/1] virtio-ism: introduce new device virtio-ism

[Wen Gu <guwen@linux.alibaba.com>]

On 2023/3/23 22:46, Halil Pasic wrote:

> On Thu,  9 Feb 2023 11:30:56 +0800
> Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
>
> > +\subsection{Device configuration layout}\label{sec:Device Types / ISM Device / Device configuration layout}
> > +
> > +\begin{lstlisting}
> > +struct virtio_ism_config {
> > +	le128 cdid;
> > +	le64 devid;
> > +	le64 chunk_size;
> > +	le64 notify_size;
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +    \item[\field{cdid}] This is used to identify the communication domain. Only
> > +        ism devices with the same \field{cdid} can communicate. A \field{cdid}
> > +        is world-wide unique in a sense that there not be another communication
> > +        domain with the same \field{cdid}.
> > +
> > +    \item[\field{devid}] This is used to identify the ism device in the single
> > +        communication domain.
> > +
> > +    \item[\field{chunk_size}] This is the size of the ism chunk. The device
> > +        memory is divided into multiple chunks. Every ism chunk has the same
> > +        size.
> > +
> > +    \item[\field{notify_size}] This is the size of the ism notify-address. The
> > +        notify-address is used to notify the device that the content of the
> > +        ism region has been updated.
> > +
> > +\end{description}
> > +
> > +\devicenormative{\subsubsection}{Device configuration layout}{Device Types / ISM Device / Device configuration layout}
> > +
> > +The device MUST ensure that the \field{cdid} of the device on the same
> > +communication domain is same. The \field{cdid} MUST be a version 4 UUID as
> > +specified by \hyperref[intro:rfc4122]{[RFC4122]}.
> > +
> > +In the single communication domain, the device MUST ensure that the \field{devid}
> > +is unique.
> > +
>
>
> Hi Xuan Zhou!
>
> My understanding is the following: you goal for virtio-ism is that
> it should be suitable for usage with SMC-D (much like the original ISM
> device). Is that right?
>
> If yes, then let us have a look at the following example. We have
> two guests sitting on the same hypervisor: A and B. Both of the
> guests have an rdma capable interface, a virtio-ism device and
> traditional ISM device. So they could talk over SMC-R, SMC-D via
> virtio and SMC-D via (PCI-)ISM. How would the CLC Proposal message
> look like?

Hi Halil!

Now SMCv2 protocol allows CLC Proposal message to carry 1 SMC-R
GID, 1 SMC-Rv2 GID, and up to 8 SMC-D [CHID,GID] at the same time
for server to choose from.

And after receiving CLC Proposal message, server will match the local
devices with devices provided in CLC Proposal messages in this order:

SMC-Dv2 (ISMv2) device -> SMC-Dv1 (ISMv1) device -> SMC-Rv2 (RoCEv2) device -> SMC-Rv1 (RoCE) device

If there are multiple SMC-Dv2 devices provided in CLC Proposal messages,
such as virtio-ism and ISM in this example, the priority depends on their
order in the slot. (we treat virtio-ism as a kind of SMC-Dv2 device)

> Where I am going with this? Either you need a novel way to
> discover peers (probably before the usual way is employed)
> or (probably preferably) you need to make this part of the
> CLC stuff. What are your ideas with regards to this? How is
> it supposed to work?

The discussion about
1) how to distinguish SMC-D device type (loopback/virtio-ism/ISM)
2) how to make sure peer virtio-ism can talk to
3) how to define priority or preference of SMC-D devices
are on-going in the mail thread of meeting note: "SMC interlock | Alibaba - IBM | 2023-03-20"

A quich summary here:

1) we propose to use reserved CHID to indicate what kind of device it is.
   such as 0xFFFF for loopback, 0xFFFE for virtio-ism.

2) whether two peer's SMC-D device can talk depends on:
   a. whether VCHID is the same;
   b. whether smc_ism_cantalk() return true;
   c. whether SEID is the same;

   In step b, smc_ism_cantalk() will simply call driver interface of
   ops->query_remote_gid() to tell. So as you can see, GID is the key
   point to check whether peers' SMC-D device can talk.

   And cdid here in virtio-ism determines whether talk is possible.
   Same cdid means two virtio-ism can talk to each other.

   So the problem may be how to covert 128-bits cdid to 64-bits GID
   used by SMC-D and avoid GID collision ? I have no elegant idea
   only a workaround right now ... The discussion on GID collision
   is now on-going in mail thread of meeting note: "SMC interlock | Alibaba - IBM | 2023-03-20".
   If you have any comments about this, very welcome to discuss there.

3) Priority or preference of SMC-D devices are decided by slot
   order now, or explicitly by CHID if we all agree to use reserved
   CHID for extension SMC-D device. (haven't been comfirmed now)

> To get back to the things proposed here: the cdid is IMHO
> a nice thing, and is functionally corresponding to the
> (S)EID. But it is 16 byte wide, and I have no idea how
> is it supposed to be used in the CLC handshake.

CLC handshake carry one SEID for all the SMC-D device. Considering
coexistence with ISM, I am not sure whether we can change or increase
the SEID.. cc Alexandra

Thanks!
Wen Gu

> If this is really supposed to work with SMC and not just take
> inspiration from it, I would like some insight from our
> SMC experts (they are already on copy).
>
> Regards,
> Halil
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org