[Users] SRP pkey support

[Blake Caldwell]

Following up on a discussion at the OFA user's day last week, I 
investigated whether ib_srpt (part of linux SCST) supported pkeys other 
than 0xffff. It doesn't, since the pkey_index is hard coded to 0 in 
srpt_init_ch_qp().  This means the target will always use pkey 0xffff. 
Discussions on this being implemented is better suited to the linux-rdma 
list, but the pkey verification and srp commands might be useful to 
others on the list.

With this in partitions.conf:
Partition=0x8004, ipoib : 0x0002c903004bfd7b=full, 0x0002c903004ae31b=full ;

I could see that opensm was using my partitions.conf and the nodes got 
the 0x8004 pkey.  I ran opensm with debug options of 0x80 (not sure this 
is necessary to get these particular messages).
Apr 23 20:45:23 435418 [41007940] 0x08 -> 
pkey_mgr_process_physical_port: pkey 0x8004 was updated for node 
0x0002c903004ae31a port 1
Apr 23 20:45:23 435422 [41007940] 0x08 -> 
pkey_mgr_process_physical_port: pkey 0x8004 was updated for node 
0x0002c903004bfd7a port 1
Apr 23 20:45:23 435425 [41007940] 0x08 -> 
pkey_mgr_process_physical_port: pkey 0x7fff was updated for node 
0x0002c903004ae31a port 1
Apr 23 20:45:23 435428 [41007940] 0x08 -> 
pkey_mgr_process_physical_port: pkey 0xffff was updated for node 
0x0002c903004bfd7a port 1

You can't use srp_daemon because it hard codes a pkey of 0xffff, so if 
the target was to support other pkeys, you'd have to do a manual srp 
login. Get the login information from ibsrpdm -c and substitute the pkey 
8004 for ffff. Log into the target with the following:
echo -n 
id_ext=[guid],ioc_guid=[guid],dgid=fe800000000000000002c903004ae31b,pkey=8004,service_id=[guid] 
 > /sys/class/infiniband_srp/srp-mlx4_1-1/add_target

Dmesg tells the story:
[5263703.357312]  host12: ib_srp: new target: id_ext [guid] ioc_guid 
[guid] pkey 8004 service_id [guid] dgid 
fe80:0000:0000:0000:0002:c903:004a:e31b
[5263703.369485] scsi12 : SRP.T10:1AE34A0003C90200
[5263708.856427]  host12: SRP abort called
[5263713.843543]  host12: SRP reset_device called
[5263718.830818]  host12: ib_srp: SRP reset_host called state 0 qp_err 0
[5263738.795576]  host12: SRP abort called
[5263743.782699]  host12: SRP reset_device called
[5263748.770978]  host12: ib_srp: SRP reset_host called state 0 qp_err 0
[5263758.761393] scsi 12:0:0:0: scsi: Device offlined - not ready after 
error recovery
[5263758.761400] scsi 12:0:0:0: timing out command, waited 22s

The same login command with pkey=ffff is successful, because pkey_index 
0 is still 0xffff.

-Blake