[ofa-general] missing "balance" in aggregate bi-directional SDP bulk transfer

[Or Gerlitz]

Rick Jones wrote:
> I configured ib0 and ib1 into separate IP subnets, and ran the 
> "bidirectional TCP_RR" test 

> However, when I run the same test over SDP, some connections seem to get 
> much better performance than others.  For example, with two concurrent 
> connections, one over each port, one will get a much higher result than 
> the other.

Did you make sure that each neighbour was actually pointing to a 
different port? see the below excerpt from the IPoIB release notes (note 
that IPoIB is the ARP provider used by the RDMA CM which is what SDP is 
working with, so this applies both your IPoIB and SDP tests).

Or.


> from /usr/share/doc/ofed-docs-1.2/ipoib_release_notes.txt
> 
> 3. Known Issues
> ===============================================================================
> 1. If a host has multiple interfaces and (a) each interface belongs to a
>    different IP subnet, (b) they all use the same InfiniBand Partition, and (c)
>    they are connected to the same IB Switch, then the host violates the IP rule
>    requiring different broadcast domains. Consequently, the host may build an
>    incorrect ARP table.
> 
>    The correct setting of a multi-homed IPoIB host is achieved by using a
>    different PKEY for each IP subnet. If a host has multiple interfaces on the
>    same IP subnet, then to prevent a peer from building an incorrect ARP entry
>    (neighbor) set the net.ipv4.conf.X.arp_ignore value to 1 or 2, where X
>    stands for the IPoIB (non-child) interfaces (e.g., ib0, ib1, etc). This
>    causes the network stack to send ARP replies only on the interface with the
>    IP address specified in the ARP request:
> 
>    sysctl -w net.ipv4.conf.ib0.arp_ignore=1
>    sysctl -w net.ipv4.conf.ib1.arp_ignore=1
> 
>    Or, globally,
> 
>    sysctl -w net.ipv4.conf.all.arp_ignore=1
> 
>    To learn more about the arp_ignore parameter, see Documentation/networking/ip-sysctl.txt.
>    Note that distributions have the means to make kernel parameters persistent.
>