[Users] IB partitioning

[Ira Weiny]

On Wed, 3 Oct 2012 13:52:34 +0200
Jesper Larsen <JLA at fcoo.dk> wrote:

> Hi IB people
> 
> I am trying to get basic partitioning working on our IB using opensm. I am using this (partitions.conf) configuration file:
> 
> Default=0x7fff, rate=7, defmember=limited: ALL;
> DevNet=0x7eee, rate=7, defmember=full: 0x78e7d1030021ecad, 0x78e7d10300245fa5;
> 
> This means that I make all members of the default group limited. And I create a new group which has two full members, namely two of our computing nodes. When I run (as root):
> 
> # ibping -S
> 
> on the first node and
> 
> # ibping -G 0x78e7d1030021ecad

I believe this may be a bug in ibping not your partition config.  But I am still trying to verify it.  (We don't use partitions around here so I have to set up a test.)

If I can get a patch together would you be willing to try?

One thing you should be able to do to verify the PKeys are getting set on the ports is to look at the pkey table.

bash-4.1# smpquery PKeys 1
   0: 0xffff 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
4 pkeys capacity for this port

smpquery PKeys <lid>

You should see 0x7fff and 0xfeee as the 2 keys.  I think the bug is that ibping is not setting the pkey index to 1 which would use your "DevNet" partition.

Also, I don't see how to set a pkey in ibv_rc_pingpong, rdma_bw, or other "low level" IB tests.  Perhaps someone else has a different test you could try in the mean time?  Or knows how to use partitions with those tests?

Ira

> 
> on the other node I get:
> 
> ibwarn: [29529] mad_rpc_rmpp: _do_madrpc failed; dport (Lid 9)
> 
> When the config file simply contains:
> 
> Default=0x7fff, rate=7, defmember=full: ALL;
> 
> the ping works fine. Does anyone know what is wrong with the first partitions.conf file?
> 
> Best regards,
> Jesper Baasch-Larsen
> _______________________________________________
> Users mailing list
> Users at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/users


-- 
Ira Weiny
Member of Technical Staff
Lawrence Livermore National Lab
925-423-8008
weiny2 at llnl.gov