Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [PATCH v13] virtio-net: support inner header hash

[Heng Qi <hengqi@linux.alibaba.com>]

On Thu, May 11, 2023 at 02:22:12AM -0400, Michael S. Tsirkin wrote:
> On Wed, May 10, 2023 at 05:15:37PM +0800, Heng Qi wrote:
> > 
> > 
> > å 2023/5/9 äå11:15, Michael S. Tsirkin åé:
> > > On Tue, May 09, 2023 at 10:22:19PM +0800, Heng Qi wrote:
> > > > 
> > > > å 2023/5/5 äå10:56, Michael S. Tsirkin åé:
> > > > > On Fri, May 05, 2023 at 09:51:15PM +0800, Heng Qi wrote:
> > > > > > On Thu, Apr 27, 2023 at 01:13:29PM -0400, Michael S. Tsirkin wrote:
> > > > > > > On Thu, Apr 27, 2023 at 10:28:29AM +0800, Heng Qi wrote:
> > > > > > > > å 2023/4/26 äå10:48, Michael S. Tsirkin åé:
> > > > > > > > > On Wed, Apr 26, 2023 at 10:14:30PM +0800, Heng Qi wrote:
> > > > > > > > > > This does not mean that every device needs to implement and support all of
> > > > > > > > > > these, they can choose to support some protocols they want.
> > > > > > > > > > 
> > > > > > > > > > I add these because we have scale application scenarios for modern protocols
> > > > > > > > > > VXLAN-GPE/GENEVE:
> > > > > > > > > > 
> > > > > > > > > > +\item In scenarios where the same flow passing through different tunnels is expected to be received in the same queue,
> > > > > > > > > > +      warm caches, lessing locking, etc. are optimized to obtain receiving performance.
> > > > > > > > > > 
> > > > > > > > > > 
> > > > > > > > > > Maybe the legacy GRE, VXLAN-GPE and GENEVE? But it has a little crossover.
> > > > > > > > > > 
> > > > > > > > > > Thanks.
> > > > > > > > > But VXLAN-GPE/GENEVE can use source port for entropy.
> > > > > > > > > 
> > > > > > > > > 	It is recommended that the UDP source port number
> > > > > > > > > 	 be calculated using a hash of fields from the inner packet
> > > > > > > > > 
> > > > > > > > > That is best because
> > > > > > > > > it allows end to end control and is protocol agnostic.
> > > > > > > > Yes. I agree with this, I don't think we have an argument on this point
> > > > > > > > right now.:)
> > > > > > > > 
> > > > > > > > For VXLAN-GPE/GENEVE or other modern tunneling protocols, we have to deal
> > > > > > > > with
> > > > > > > > scenarios where the same flow passes through different tunnels.
> > > > > > > > 
> > > > > > > > Having them hashed to the same rx queue, is hard to do via outer headers.
> > > > > > > > > All that is missing is symmetric Toepliz and all is well?
> > > > > > > > The scenarios above or in the commit log also require inner headers.
> > > > > > > Hmm I am not sure I get it 100%.
> > > > > > > Could you show an example with inner header hash in the port #,
> > > > > > > hash is symmetric, and you still have trouble?
> > > > > > > 
> > > > > > > 
> > > > > > > It kinds of sounds like not enough entropy is not the problem
> > > > > > > at this point.
> > > > > > Sorry for the late reply. :)
> > > > > > 
> > > > > > For modern tunneling protocols, yes.
> > > > > > 
> > > > > > > You now want to drop everything from the header
> > > > > > > except the UDP source port. Is that a fair summary?
> > > > > > > 
> > > > > > For example, for the same flow passing through different VXLAN tunnels,
> > > > > > packets in this flow have the same inner header and different outer
> > > > > > headers. Sometimes these packets of the flow need to be hashed to the
> > > > > > same rxq, then we can use the inner header as the hash input.
> > > > > > 
> > > > > > Thanks!
> > > > > So, they will have the same source port yes?
> > > > Yes. The outer source port can be calculated using the 5-tuple of the
> > > > original packet,
> > > > and the outer ports are the same but the outer IPs are different after
> > > > different directions of the same flow pass through different tunnels.
> > > > > Any way to use that
> > > > We use it in monitoring, firewall and other scenarios.
> > > > 
> > > > > so we don't depend on a specific protocol?
> > > > Yes, selected tunneling protocols can be used in this scenario like this.
> > > > 
> > > > Thanks.
> > > > 
> > > No, the question was - can we generalize this somehow then?
> > > For example, a flag to ignore source IP when hashing?
> > > Or maybe just for UDP packets?
> > 
> > 1. I think the common solution is based on the inner header, so that
> > GRE/IPIP tunnels can also enjoy inner symmetric hashing.
> > 
> > 2. The VXLAN spec does not show that the outer source port in both
> > directions of the same flow must be the same [1]
> > (although the outer source port is calculated based on the consistent hash
> > in the kernel. The consistent hash will sort the five-tuple before
> > calculating hashing),
> > but it is best not to assume that consistent hashing is used in all VXLAN
> > implementations.
> 
> I agree, best not to assume if it's not in the spec.
> The requirement to hash two sides to same queue might
> not be necessary for everyone though, right?

The outer source port is also not reliable when it needs to be hashed to
the same queue, but the inner header identifies a flow reliably and
universally.

> 
> > The GENEVE spec uses "SHOUlD"[2].
> 
> What about other tunnels? Could you summarize please?

Sure.

The VXLAN spec[1] does not show that the outer source port in both
directions of the same flow must be the same.

VXLAN-GPE[2]("SHOULD")/GENEVE[3]("SHOULD")/GRE-in-UDP[4.1]/STT[5]
recommend that the outer source port of the same flow be calculated
based on the inner header hash and set to the same.

But the udp source port of GRE-in-UDP may be used in a scenario similar
to NAPT [4.2], where the udp source port is no longer used for entropy,
but for identifying different internal hosts. So using udp source port
does not identify the same stream. This is why using the inner header is
more general, since information about the original stream can reliably
identify a flow.

[1] "Source Port: It is recommended that the UDP source port number be
calculated using a hash of fields from the inner packet -- one example
being a hash of the inner Ethernet frame's headers. This is to enable a
level of entropy for the ECMP/load-balancing of the VM-to-VM traffic
across the VXLAN overlay. When calculating the UDP source port number in
this manner, it is RECOMMENDED that the value be in the dynamic/private
port range 49152-65535 [RFC6335]"

[2] "Source UDP Port: The source UDP port is used as entropy for devices
forwarding encapsulated packets across the underlay (ECMP for IP routers,
or load splitting for link aggregation by bridges). Tenant traffic flows
should all use the same source UDP port to lower the chances of packet
reordering by the underlay for a given flow. It is recommended for VTEPs
to generate this port number using a hash of the inner packet headers.
Implementations MAY use the entire 16 bit source UDP port for entropy."

[3] "Source Port: A source port selected by the originating tunnel
endpoint. This source port SHOULD be the same for all packets belonging
to a single encapsulated flow to prevent reordering due to the use of
different paths. To encourage an even distribution of flows across
multiple links, the source port SHOULD be calculated using a hash of the
encapsulated packet headers using, for example, a traditional 5-tuple.
Since the port represents a flow identifier rather than a true UDP
connection, the entire 16-bit range MAY be used to maximize entropy."

[4.1] "GRE-in-UDP permits the UDP source port value to be used to encode
an entropy value. The UDP source port contains a 16-bit entropy value
that is generated by the encapsulator to identify a flow for the
encapsulated packet. The port value SHOULD be within the ephemeral port
range, i.e., 49152 to 65535, where the high-order two bits of the port
are set to one. This provides fourteen bits of entropy for the inner
flow identifier. In the case that an encapsulator is unable to derive
flow entropy from the payload header or the entropy usage has to be
disabled to meet operational requirements (see Section 7), to avoid
reordering with a packet flow, the encapsulator SHOULD use the same UDP
source port value for all packets assigned to a flow, e.g., the result
of an algorithm that performs a hash of the tunnel ingress and egress IP
address."

[4.2] "use of the UDP source port for entropy may impact middleboxes'
behavior. If a GRE-in-UDP tunnel is expected to be used on a path
with a middlebox, the tunnel can be configured either to disable use
of the UDP source port for entropy or to enable middleboxes to pass
packets with UDP source port entropy."

[5] "STT achieves the first goal by ensuring that the source and
destination ports and addresses in the outer header are all the same for
a single flow.  The second goal is achieved by generating the source
port using a random hash of fields in the headers of the inner packets,
e.g. the ports and addresses of the virtual flow's packets."

> SHOULD means "if you ignore this
> things will work but not well".
> You mentioned concerns such as worse performance,
> this is fine with SHOULD.

That's it.

> Is inner hashing important for
> correctness sometimes?

I'm sorry I didn't understand this, can you explain it in more detail?

> 
> > 3. How should we generalize? The device uses a feature to advertise all the
> > tunnel types it supports, and hashes these tunnel types using the outer
> > source port,
> > and then we still have to give the specific tunneling protocols supported by
> > the device, just like we do now.
> 
> Is it problematic to do this for all UDP packets?

I think there will be problems. While devices support configuring this,
drivers sometimes don't want devices to do special handling for certain
tunneling protocols.

Thanks.

> 
> > [1] "Source Port: It is recommended that the UDP source port number be
> > calculated using a hash of fields from the inner packet -- one example
> > being a hash of the inner Ethernet frame's headers. This is to enable a
> > level of entropy for the ECMP/load-balancing of the VM-to-VM traffic across
> > the VXLAN overlay. When calculating the UDP source port number in this
> > manner, it is RECOMMENDED that the value be in the dynamic/private
> > port range 49152-65535 [RFC6335] "
> > 
> > [2] "Source Port: A source port selected by the originating tunnel endpoint.
> > This source port SHOULD be the same for all packets belonging to a
> > single encapsulated flow to prevent reordering due to the use of different
> > paths. To encourage an even distribution of flows across multiple links,
> > the source port SHOULD be calculated using a hash of the encapsulated packet
> > headers using, for example, a traditional 5-tuple. Since the port
> > represents a flow identifier rather than a true UDP connection, the entire
> > 16-bit range MAY be used to maximize entropy. In addition to setting the
> > source port, for IPv6, the flow label MAY also be used for providing
> > entropy. For an example of using the IPv6 flow label for tunnel use cases,
> > see [RFC6438]."
> > 
> > Thanks.
> > 
> > > 
> > 
> > 
> > This publicly archived list offers a means to provide input to the
> > OASIS Virtual I/O Device (VIRTIO) TC.
> > 
> > In order to verify user consent to the Feedback License terms and
> > to minimize spam in the list archive, subscription is required
> > before posting.
> > 
> > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > List help: virtio-comment-help@lists.oasis-open.org
> > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > Committee: https://www.oasis-open.org/committees/virtio/
> > Join OASIS: https://www.oasis-open.org/join/
> >