[Users] IB Partitioning

[Jesper Larsen]

Dear OFED users,

We have a cluster connected by a single IB switch which we want to split into two separate partitions (development/test and production). The partitions must not be able to talk to each other since we do not want errors happening on the development partition to be able to affect the production partition. I have therefore tried to make a partition configuration file for opensm:

/etc/opensm/partitions.conf

which at this point (just testing if I can make it work for a single non-default partition) looks something like this:

Default=0xffff, rate=7, defmember=limited: ALL;
DevNet=0x0001, ipoib, rate=7, defmember=full: 0x0002c903004ef895, 0x78e7d1030023ffd5, 0x78e7d1030023fdfd, 0x78e7d10300239885, 0x78e7d1030023ff7d, 0x78e7d1030021ecad, 0x78e7d10300245fa5, 0x78e7d1030024578d;

So I have made all the members of the default group limited so that they cannot talk together. The reason is that all nodes are members of the default group and that cannot as far as I understand be changed. And I have made a new group containing only development nodes. But when I try to ping one development node from another (over ib0) it cannot find it (Destination Host Unreachable). And when I look at at the two development nodes I see:

# cat /sys/class/net/ib0/pkey 
0xffff

Does this mean that it is only trying the default network? And if yes, how do I make it try the DevNet?

Best regards,
Jesper