[openib-general] Re: IPoIB still not working

[Roland Dreier]

    Doug> Well, he had identical hardware addresses on n0 and n1 for
    Doug> his ib0 and ib1 interfaces respectively.  This would likely
    Doug> keep the linux network stack from actually sending the
    Doug> packet back to the originating host and instead convince it
    Doug> that it's intended for internal consumption I would think.
    Doug> Try putting a different HW MAC address on your different ib?
    Doug> ports and see if that gets the packets flowing.

Actually ifconfig can only show the first 16 octets of the HW address
(and I think the last two bytes are actually wrong, because the
SIOGIFHWADDR ioctl that it uses can only return 14 bytes).  IPoIB has
a 20 byte HW address; the four (or six?) bytes that get cut off are
the low-order bytes of the port GID, which is probably where the
difference between port GIDs is.

To see the real address, you need to do something like "ip addr show
dev ib0".  For example, on my system:

    # ifconfig ib0
    ib0       Link encap:UNSPEC  HWaddr 00-00-04-04-FE-80-00-00-00-00-00-00-00-00-00-00
              BROADCAST MULTICAST  MTU:2044  Metric:1
              RX packets:0 errors:0 dropped:0 overruns:0 frame:0
              TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:128
              RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

    # ip addr show dev ib0
    5: ib0: <BROADCAST,MULTICAST> mtu 2044 qdisc noop qlen 128
        link/[32] 00:00:04:04:fe:80:00:00:00:00:00:00:00:02:c9:01:07:8c:e4:61 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff

Perhaps we need something about this in the growing IPoIB FAQ?

 - Roland