[ofw] enable/disable device changes IPoIB physical address??

[Fab Tillier]

Hi Jeremy,

In Windows, IPoIB pretends to be a standard 802.3 NIC.  That means the
MAC address it reports to the OS for both its local address and in any
received ARP packets is a 6-byte MAC.  This 6-byte MAC is generated from
the 20-byte IPoIB MAC that goes over the wire and is made up of the GID
and QPN.  Shrinking a 20-byte space down to 6 bytes is a bit of a
challenge.  The driver discards the QPN and the subnet prefix from the
GID, leaving just the port GUID.  This GUID is then reduced to a 6-byte
MAC based on the vendor ID in the first 3 bytes.  There are algorithms
in the IPoIB driver to convert Mellanox, QLogic (ex Sliverstorm), and
Voltaire GUIDs.  For all other types, the driver generates a locally
administered address (LAA) which starts with 02 00 (maybe an extra 00,
too).  The lower bytes are a simple counter that gets incremented every
time a new LAA is generated.  Note that Cisco HCAs fall into the "all
other types" category, as there is currently no code in the IPoIB driver
to handle these GUIDs.

When the driver first comes up, on a dual port HCA the first port gets
02 00 00 00 00 01, and the second 02 00 00 00 00 02.  For every received
ARP it allocates another LAA.  If you disable the first instance (the
one that had 02 00 00 00 00 01 assigned) and then re-enable it, it will
get the next value.  That value depends on how many LAAs have been
allocated, so can vary.  Note that because you have the second IPoIB
instance (02 00 00 00 00 02) enabled, the driver stays in memory when
you disable/enable the first.  The driver will be unloaded when the last
instance is disabled, but not otherwise.  If you keep the second
instance disabled, the driver will be unloaded when you disable the
first, and when you re-enable the driver starts fresh assigning LAAs.

So: disable the second instance and your script will work fine without
the reboots.

As to the other problem of assigning IP addresses I haven't seen that
behavior so I'm useless here.

Cheers,
-Fab



-----Original Message-----
From: ofw-bounces at lists.openfabrics.org
[mailto:ofw-bounces at lists.openfabrics.org] On Behalf Of Jeremy Enos
Sent: Wednesday, August 01, 2007 1:16 PM
To: ofw at lists.openfabrics.org; David A. Norris
Subject: [ofw] enable/disable device changes IPoIB physical address??

Hi-
We're attempting to create some automated scripts to change IP settings 
w/ IPoIB adapters, and running into some strange issues.  Thus far, 
we've been trying to identify the correct NIC/port by the physical 
address, which seems to be the same on all hosts.  This is fine, as we 
were looking for a reliable way to identify the port each time.  
However, it's *not* reliable.  For some reason, disabling and 
re-enabling the device actually changes the physical address that 
ipconfig reports each time.  Only a reboot of the host resets it to the 
initial address (02-00-00-00-00-01).  We're also guessing that the phys 
addy that gets assigned on each re-enabling of the device is based on 
some timestamp, as the disparity between them increases when there is 
more time between the re-enabling.  Why is this address ever changing in

the first place?

Environment:
OS: Win2003 x64
HCA:  MT25208

Another explanation I'm seeking is why setting static IP info via the 
GUI doesn't seem to take.  No errors when I hit "Ok", but when I go back

to the tcp/ip settings again, it's all reset to dhcp.  Even when command

line utilities like netsh are used to change the IP from the cmd line, 
the GUI does not reflect it.  Anyone else see this?

thx-

    Jeremy Enos
    NCSA

_______________________________________________
ofw mailing list
ofw at lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw