[Users] IB partitioning

[Jesper Larsen]

Ira Weiny [weiny2 at llnl.gov] wrote:
>I believe this may be a bug in ibping not your partition config.  But I am still trying to verify it.  (We don't use partitions around here so I have to set up a test.)
>
>If I can get a patch together would you be willing to try?

Definitely.

>One thing you should be able to do to verify the PKeys are getting set on the ports is to look at the pkey table.
>
>bash-4.1# smpquery PKeys 1
>   0: 0xffff 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
>4 pkeys capacity for this port
>
>smpquery PKeys <lid>
>
>You should see 0x7fff and 0xfeee as the 2 keys.  I think the bug is that ibping is not setting the pkey index to 1 which would use your "DevNet" partition.

The two nodes in my post have lids 9 and 13:

[root at bifrost opensm]# smpquery PKeys 9
   0: 0x7fff 0xfeee 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
   8: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
  16: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
  24: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
  32: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
  40: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
  48: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
  56: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
  64: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
  72: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
  80: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
  88: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
  96: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
 104: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
 112: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
 120: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
128 pkeys capacity for this port
[root at bifrost opensm]# smpquery PKeys 13
   0: 0x7fff 0xfeee 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
   8: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
  16: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
  24: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
  32: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
  40: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
  48: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
  56: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
  64: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
  72: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
  80: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
  88: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
  96: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
 104: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
 112: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
 120: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
128 pkeys capacity for this port

So both keys are present.

I am not sure how pkeys work. How is the proper pkey selected for communication between two nodes? Do I have to be concerned about that or should ibping (or some lower level libraries) take care of it?

>Also, I don't see how to set a pkey in ibv_rc_pingpong, rdma_bw, or other "low level" IB tests.  Perhaps someone else has a different test you could try in the mean time?  Or knows how to use partitions with those tests?

I also tried testing an IPoIB with partitions without luck (with a plain ping command). I wrote about that in an earlier thread. Without any success. So I tried to make this test simpler.

>Ira

Thanks for your input so far.

Jesper