[Users] IB Partitioning

[David McMillen]

On Fri, Sep 14, 2012 at 4:21 AM, Jesper Larsen <JLA at fcoo.dk> wrote:

> Dear OFED users,
>
> We have a cluster connected by a single IB switch which we want to split
> into two separate partitions (development/test and production). The
> partitions must not be able to talk to each other since we do not want
> errors happening on the development partition to be able to affect the
> production partition. I have therefore tried to make a partition
> configuration file for opensm:
>
> /etc/opensm/partitions.conf
>
> which at this point (just testing if I can make it work for a single
> non-default partition) looks something like this:
>
> Default=0xffff, rate=7, defmember=limited: ALL;
> DevNet=0x0001, ipoib, rate=7, defmember=full: 0x0002c903004ef895,
> 0x78e7d1030023ffd5, 0x78e7d1030023fdfd, 0x78e7d10300239885,
> 0x78e7d1030023ff7d, 0x78e7d1030021ecad, 0x78e7d10300245fa5,
> 0x78e7d1030024578d;
>
> So I have made all the members of the default group limited so that they
> cannot talk together. The reason is that all nodes are members of the
> default group and that cannot as far as I understand be changed. And I have
> made a new group containing only development nodes. But when I try to ping
> one development node from another (over ib0) it cannot find it (Destination
> Host Unreachable). And when I look at at the two development nodes I see:
>
> # cat /sys/class/net/ib0/pkey
> 0xffff
>

This will always show 0xffff since ib0 is always on the default partition.

Your 0x0001 partition will be used by ib0.8001 since IB partitions look
like VLANs in the network interface world.

If you want to know what pkeys your HCA has, look at something
like /sys/class/infiniband/*/ports/1/pkeys/*

Dave



>
> Does this mean that it is only trying the default network? And if yes, how
> do I make it try the DevNet?
>
> Best regards,
> Jesper
> _______________________________________________
> Users mailing list
> Users at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/users/attachments/20120914/8384ebab/attachment.html>