Re: [virtio-comment] [PATCH] virtio-net: Add support for the flexible driver notification

["Michael S. Tsirkin" <mst@redhat.com>]

On Tue, Jan 28, 2020 at 02:33:27PM +0000, Vitaly Mireyno wrote:
> 
> >-----Original Message-----
> >From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
> >open.org> On Behalf Of Michael S. Tsirkin
> >Sent: Monday, 27 January, 2020 20:48
> >To: Vitaly Mireyno <vmireyno@marvell.com>
> >Cc: Stefan Hajnoczi <stefanha@redhat.com>; virtio-comment@lists.oasis-
> >open.org; Jason Wang <jasowang@redhat.com>
> >Subject: [EXT] Re: [virtio-comment] [PATCH] virtio-net: Add support for the
> >flexible driver notification
> >
> >On Mon, Jan 27, 2020 at 05:00:03PM +0000, Vitaly Mireyno wrote:
> >>
> >> >-----Original Message-----
> >> >From: Michael S. Tsirkin <mst@redhat.com>
> >> >Sent: Monday, 27 January, 2020 17:34
> >> >To: Vitaly Mireyno <vmireyno@marvell.com>
> >> >Cc: Stefan Hajnoczi <stefanha@redhat.com>;
> >> >virtio-comment@lists.oasis- open.org; Jason Wang
> >> ><jasowang@redhat.com>
> >> >Subject: [EXT] Re: [virtio-comment] [PATCH] virtio-net: Add support
> >> >for the flexible driver notification
> >> >
> >> >On Mon, Jan 27, 2020 at 12:49:48PM +0000, Vitaly Mireyno wrote:
> >> >>
> >> >> >-----Original Message-----
> >> >> >From: Stefan Hajnoczi <stefanha@redhat.com>
> >> >> >Sent: Monday, 27 January, 2020 11:50
> >> >> >To: Vitaly Mireyno <vmireyno@marvell.com>
> >> >> >Cc: virtio-comment@lists.oasis-open.org; Michael S. Tsirkin
> >> >> ><mst@redhat.com>; Jason Wang <jasowang@redhat.com>
> >> >> >Subject: [EXT] Re: [virtio-comment] [PATCH] virtio-net: Add
> >> >> >support for the flexible driver notification
> >> >> >
> >> >> >On Sun, Jan 26, 2020 at 12:10:28PM +0000, Vitaly Mireyno wrote:
> >> >> >> Currently, the driver notification (available buffer
> >> >> >> notification) has a fixed
> >> >> >structure.
> >> >> >> If VIRTIO_F_NOTIFICATION_DATA has been negotiated, it includes:
> >> >> >> vqn,
> >> >> >next_off and next_wrap.
> >> >> >> If notify_off_multiplier > 0, the VQ number can be derived by
> >> >> >> the device
> >> >> >from the Queue Notify address, so vqn may be redundant.
> >> >> >>
> >> >> >> Some devices benefit from receiving an additional data with
> >> >> >> driver
> >> >> >notifications (because of the specific hardware implementation).
> >> >> >This data can optionally replace the vqn field in the driver notification
> >structure.
> >> >> >> In its simplest form, it would be sufficient for this data to be
> >> >> >> a per-device
> >> >> >constant value.
> >> >> >>
> >> >> >>
> >> >> >> Signed-off-by: Vitaly Mireyno <vmireyno@marvell.com>
> >> >> >> ---
> >> >> >>  content.tex | 24 ++++++++++++++++++++++++
> >> >> >>  1 file changed, 24 insertions(+)
> >> >> >
> >> >> >Hi Vitaly,
> >> >> >Can you give a concrete example of how devices can use this feature?
> >> >> >
> >> >> >The reason I'm a little unsure is that the PCI capability only
> >> >> >provides a single notify_data value per device.  That same value
> >> >> >is then passed back to the device in the notify write.
> >> >> >
> >> >> >If there is just a single constant value per device, why is it
> >> >> >necessary to involve the driver at all?  Doesn't the device
> >> >> >already know its own magic value and it doesn't really need the
> >> >> >driver to pass the
> >> >value back to it?
> >> >> >
> >> >> >Stefan
> >> >> >
> >> >>
> >> >> This will help HW devices that share the same memory space between
> >> >virtqueue notifications and other types of notifications. The virtio
> >> >driver will always use the value that will tell the HW that it's a virtqueue
> >notification.
> >> >> Of course, the feature can be extended, such that there will be a
> >> >> value
> >> >configurable per virtqueue (which is a generalization of the vqn
> >> >field). I just didn't want to complicate it further, without a real-life
> >justification.
> >> >
> >> >I would like to see a bit more detail about these other types of
> >> >notifications then.
> >> >
> >>
> >> Other notification types could be control-plane notifications, for example
> >adding MAC/VLAN filter. These types of notifications will not be sent by the
> >virtio driver, but rather by a vendor driver in the vDPA framework.
> >> Another example is offloaded RDMA notifications.
> >> The thing is that the HW doesn't have a "mode" configuration that can
> >distinguish between virtio and other modes. The HW only reacts to the
> >notification data, which is expected to include all the relevant information.
> >
> >If it's control plane, isn't the natural way to do this is by mapping a separate
> >range of addresses though?
> >
> >It might be a good idea to add extra data in the notifications but wasting these
> >bits on control plane things seems like a weird design choice.
> >
> 
> You are right, but the control plane is only an example. It's mainly
> designed to be used for other offloaded protocols data plane
> notifications, and since the HW infrastructure is already there, the
> control plane may as well use it.

OK so I wonder what are some other examples? And are these well served
by a static key in this field?

> Sharing the same address range
> helps reducing the PCI BAR size.

OTOH is there a need to have this per VQ? Can't control plane things
just use a single config space register? This way there's no real
increase in the BAR size.

> >
> >> >
> >> >>
> >> >> >> diff --git a/content.tex b/content.tex index 06bb4ca..265dc12
> >> >> >> 100644
> >> >> >> --- a/content.tex
> >> >> >> +++ b/content.tex
> >> >> >> @@ -965,6 +965,9 @@ \subsubsection{Notification structure
> >> >> >> layout}\label{sec:Virtio Transport Options  struct virtio_pci_notify_cap
> >{
> >> >> >>          struct virtio_pci_cap cap;
> >> >> >>          le32 notify_off_multiplier; /* Multiplier for
> >> >> >> queue_notify_off. */
> >> >> >> +        le16 notify_data; /* The data to be placed in the vqn field */
> >> >> >> +        u8 notify_data_valid;
> >> >> >> +        u8 padding; /* Pad to a dword */
> >> >> >>  };
> >> >> >>  \end{lstlisting}
> >> >> >>
> >> >> >> @@ -984,6 +987,9 @@ \subsubsection{Notification structure
> >> >> >> layout}\label{sec:Virtio Transport Options  the same Queue
> >> >> >> Notify address
> >> >> >for all queues.
> >> >> >>  \end{note}
> >> >> >>
> >> >> >> +If \field{notify_data_valid} != 0, the driver will set the
> >> >> >> +\field{vqn} field in the available buffer notification
> >> >> >> +structure to the
> >> >> >\field{notify_data} value.
> >> >> >> +
> >> >> >>  \devicenormative{\paragraph}{Notification capability}{Virtio
> >> >> >> Transport Options / Virtio Over PCI Bus / PCI Device Layout /
> >> >> >> Notification
> >> >> >capability}  The device MUST present at least one notification capability.
> >> >> >>
> >> >> >> @@ -1020,6 +1026,12 @@ \subsubsection{Notification structure
> >> >> >> layout}\label{sec:Virtio Transport Options  cap.length >=
> >> >> >> queue_notify_off * notify_off_multiplier + 4  \end{lstlisting}
> >> >> >>
> >> >> >> +The device MAY set \field{notify_data_valid} to any non-zero
> >> >> >> +value, to request the driver to set the \field{vqn} to the
> >> >\field{notify_data} value.
> >> >> >> +If the device sets \field{notify_data_valid} to a non-zero
> >> >> >> +value, it MUST set \field{notify_data} to a valid value.
> >> >> >> +For a normal operation, the device MUST set
> >> >> >> +\field{notify_data_valid} to
> >> >> >zero.
> >> >> >> +
> >> >> >>  \subsubsection{ISR status capability}\label{sec:Virtio
> >> >> >> Transport Options / Virtio Over PCI Bus / PCI Device Layout /
> >> >> >> ISR status capability}
> >> >> >>
> >> >> >>  The VIRTIO_PCI_CAP_ISR_CFG capability @@ -1519,6 +1531,18 @@
> >> >> >> \subsubsection{Available Buffer Notifications}\label{sec:Virtio
> >> >> >> Transport Option  See \ref{sec:Virtio Transport Options / Virtio
> >> >> >> Over PCI Bus / PCI Device Layout / Notification capability}  for
> >> >> >> how to calculate the
> >> >> >Queue Notify address.
> >> >> >>
> >> >> >> +\drivernormative{\paragraph}{Available Buffer
> >> >> >> +Notifications}{Virtio Transport Options / Virtio Over PCI Bus /
> >> >> >> +PCI-specific Initialization And Device Operation / Available
> >> >> >> +Buffer Notifications} If VIRTIO_F_NOTIFICATION_DATA has been
> >> >> >> +negotiated, and if
> >> >> >\field{notify_data_valid} != 0, the driver MUST set the
> >> >> >\field{vqn} field of the available buffer notification structure
> >> >> >to the \field{notify_data}
> >> >value.
> >> >> >> +
> >> >> >> +\begin{note}
> >> >> >> +If \field{notify_off_multiplier} > 0, the virtqueue number can
> >> >> >> +potentially be derived by the device from the Queue Notify
> >> >> >> +address, so \field{vqn} may be redundant. Some devices benefit
> >> >> >> +from receiving the additional data with driver notifications
> >> >> >> +(because of the specific
> >> >> >hardware implementation).
> >> >> >> +\end{note}
> >> >> >> +
> >> >> >>  \subsubsection{Used Buffer Notifications}\label{sec:Virtio
> >> >> >> Transport Options / Virtio Over PCI Bus / PCI-specific
> >> >> >> Initialization And Device Operation / Used Buffer Notifications}
> >> >> >>
> >> >> >>  If a used buffer notification is necessary for a virtqueue, the
> >> >> >> device would
> >> >> >typically act as follows:
> >> >> >> --
> >> >> >>
> >> >> >> This publicly archived list offers a means to provide input to
> >> >> >> the OASIS Virtual I/O Device (VIRTIO) TC.
> >> >> >>
> >> >> >> In order to verify user consent to the Feedback License terms
> >> >> >> and to minimize spam in the list archive, subscription is
> >> >> >> required before posting.
> >> >> >>
> >> >> >> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> >> >> >> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> >> >> >> List help: virtio-comment-help@lists.oasis-open.org
> >> >> >> List archive:
> >> >> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.oasis
> >> >> >> -2D
> >> >> >> open.org_archives_virtio-
> >> >2Dcomment_&d=DwIFAg&c=nKjWec2b6R0mOyPaz7xt
> >> >> >>
> >>
> >>fQ&r=lDHJ2FW52oJ3lqqsArgFRdcevq01tbLQAw4A_NO7xgI&m=V_WFWAet
> >W
> >> >bHu0u2N
> >> >> >>
> >>
> >>hwNHEqFKyE_y6sUMeJcHqURDeoA&s=xTzOJ9VTGfxRUMJf0A4unV7kB_hd
> >Ml
> >> >oGKMKrQ
> >> >> >> hnSJKc&e=
> >> >> >> Feedback License:
> >> >> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.oasis-
> >> >2Dop
> >> >> >> en.org_who_ipr_feedback-
> >> >5Flicense.pdf&d=DwIFAg&c=nKjWec2b6R0mOyPaz7
> >> >> >>
> >>
> >>xtfQ&r=lDHJ2FW52oJ3lqqsArgFRdcevq01tbLQAw4A_NO7xgI&m=V_WFWAe
> >t
> >> >WbHu0u
> >> >> >>
> >>
> >>2NhwNHEqFKyE_y6sUMeJcHqURDeoA&s=vL3tJ_XMSNJwV8gDnLft9liDG6oS
> >V
> >> >HLNhdd
> >> >> >> MUe4ZZNI&e=
> >> >> >> List Guidelines:
> >> >> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.oasis-
> >> >2Dop
> >> >> >> en.org_policies-2Dguidelines_mailing-
> >> >2Dlists&d=DwIFAg&c=nKjWec2b6R0
> >> >> >>
> >>
> >>mOyPaz7xtfQ&r=lDHJ2FW52oJ3lqqsArgFRdcevq01tbLQAw4A_NO7xgI&m=V
> >_
> >> >WFWAe
> >> >> >>
> >>
> >>tWbHu0u2NhwNHEqFKyE_y6sUMeJcHqURDeoA&s=A444Hd_Yb4aycXM5Ids
> >B
> >> >41E4xDmF
> >> >> >> M7c3G7m8tQZCvBg&e=
> >> >> >> Committee:
> >> >> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.oasis-
> >> >2Dop
> >> >> >>
> >>
> >>en.org_committees_virtio_&d=DwIFAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=lD
> >> >HJ2
> >> >> >>
> >>
> >>FW52oJ3lqqsArgFRdcevq01tbLQAw4A_NO7xgI&m=V_WFWAetWbHu0u2Nh
> >w
> >> >NHEqFKyE
> >> >> >>
> >>
> >>_y6sUMeJcHqURDeoA&s=VA420pM5tOIIqUWH7lCMCkm2zbEbtBBOvotp9g
> >W
> >> >r9QM&e=
> >> >> >> Join OASIS:
> >> >> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.oasis-
> >> >2Dop
> >> >> >>
> >>
> >>en.org_join_&d=DwIFAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=lDHJ2FW52oJ3lq
> >q
> >> >sAr
> >> >> >>
> >>
> >>gFRdcevq01tbLQAw4A_NO7xgI&m=V_WFWAetWbHu0u2NhwNHEqFKyE_y6
> >s
> >> >UMeJcHqUR
> >> >> >> DeoA&s=hVpH-MUm5o9XpvEa6BOWraE8-45dGcCFvBhGRITH_nE&e=
> >> >> >>
> >
> >
> >This publicly archived list offers a means to provide input to the OASIS Virtual
> >I/O Device (VIRTIO) TC.
> >
> >In order to verify user consent to the Feedback License terms and to minimize
> >spam in the list archive, subscription is required before posting.
> >
> >Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> >Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> >List help: virtio-comment-help@lists.oasis-open.org
> >List archive: https://urldefense.proofpoint.com/v2/url?u=https-
> >3A__lists.oasis-2Dopen.org_archives_virtio-
> >2Dcomment_&d=DwIFAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=lDHJ2FW52oJ3lq
> >qsArgFRdcevq01tbLQAw4A_NO7xgI&m=ZwtynokTxS59JQ6NCjDu8EpKsv1ZL-
> >7pzZ7LKd4663I&s=6ej9qQAg0jeBVJNhzfumR3Q7eLhFNNWLpz5Zlk9KIVM&e=
> >Feedback License: https://urldefense.proofpoint.com/v2/url?u=https-
> >3A__www.oasis-2Dopen.org_who_ipr_feedback-
> >5Flicense.pdf&d=DwIFAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=lDHJ2FW52oJ3lqq
> >sArgFRdcevq01tbLQAw4A_NO7xgI&m=ZwtynokTxS59JQ6NCjDu8EpKsv1ZL-
> >7pzZ7LKd4663I&s=ivzoy5NStwJg3_x8GzB-XNSOykSYt1Sy7Yr9w4JZ7ts&e=
> >List Guidelines: https://urldefense.proofpoint.com/v2/url?u=https-
> >3A__www.oasis-2Dopen.org_policies-2Dguidelines_mailing-
> >2Dlists&d=DwIFAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=lDHJ2FW52oJ3lqqsArgFR
> >dcevq01tbLQAw4A_NO7xgI&m=ZwtynokTxS59JQ6NCjDu8EpKsv1ZL-
> >7pzZ7LKd4663I&s=Qi0rg76WzE90Oienx5UWxgIAWu27yejzprteF-WguqM&e=
> >Committee: https://urldefense.proofpoint.com/v2/url?u=https-
> >3A__www.oasis-
> >2Dopen.org_committees_virtio_&d=DwIFAg&c=nKjWec2b6R0mOyPaz7xtfQ&
> >r=lDHJ2FW52oJ3lqqsArgFRdcevq01tbLQAw4A_NO7xgI&m=ZwtynokTxS59JQ6
> >NCjDu8EpKsv1ZL-
> >7pzZ7LKd4663I&s=N9LxnUJBE6Ev1UGHjD8vbFhIWML8NF5PnVVS7N2KYSE&e=
> >Join OASIS: https://urldefense.proofpoint.com/v2/url?u=https-
> >3A__www.oasis-
> >2Dopen.org_join_&d=DwIFAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=lDHJ2FW52o
> >J3lqqsArgFRdcevq01tbLQAw4A_NO7xgI&m=ZwtynokTxS59JQ6NCjDu8EpKsv1
> >ZL-
> >7pzZ7LKd4663I&s=1euFBF86OgTr6gTGn0dUY8oY8bh6DqUoWGX4ag2Kcs0&e=
> 
> 
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
> 
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
> 
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/