Abstract: <br> We are developing SRP target code and testing it with the OFED 1.1 SRP initiator. The OFED SRP initiator sends us a CM REQ (IB 12.6.5) and we respond with CM REP (12.6.8). However, instead of the expected CM RTU (12.6.9) we ALWAYS receive a CM REJ (12.6.7) with status 0x1C == Reason 28 (12.6.7.2). <br> <br> Software Setup: <br> - SUSE 10.0 <br> - OFED 1.1 <br> - Mellanox card with 3.5.00 firmware <br> <br> Details: <br> Initially we suspected our response values in the CM REP packet. There is nothing obvious to us, and anything we weren't sure about we tried a number of combinations. This applies to the SRP private data as well. <br> We also took a look at .<i class="moz-txt-slash"><span class="moz-txt-tag">/</span>openib-1.1/drivers/infiniband/core<span class="moz-txt-tag">/</span></i> cm.c and cma.c. The function cma.c:cma_rep_recv looked like a possibility but there's little debug output.
Aside from putting in printk's and recompiling and installing is there an easier way to debug? <br> <br> Included below is the output from the initiator and packet dumps of the 3 packets.<br> <br> -------------------------------------- <br> <br> ./ibsrpdm -cv <br> id_ext=200601045300030B,ioc_guid=00045381300030b2,dgid=fe8000000000000000045381300030b2,pkey=ffff,service_id=200601045300030b <br> id_ext=2000020453000011,ioc_guid=00045381300030b2,dgid=fe8000000000000000045381300030b2,pkey=ffff,service_id=2000020453000011 <br> <br> cycl-247:/usr/local/ofed/sbin # echo id_ext=2000020453000011,ioc_guid=00045381300030b2,dgid=fe8000000000000000045381300030b2,pkey=ffff,service_id=2000020453000011 >/sys/class/infiniband_srp/srp-mthca0-1/add_target <br> <br> -------------------------------------- <br> <br> *** CM REQ *** <br> <br> received MAD [QP1]: <br> struct HdrLRH (8 bytes) - <br>
Local Route Header (section 7.7) <br> { <br> VL: 0x0 (4 bit uint) <br> LVer: 0x0 (4 bit uint) <br> SL: 0x0 (4 bit uint) <br> rsv0: 0x0 (2 bit uint) <br> LNH: 0x2 (2 bit uint) <br>
DLID: 0x0004 (16 bit uint) <br> rsv1: 0x00 (5 bit uint) <br> pktLen: 0x048 (11 bit uint) <br> SLID: 0x0104 (16 bit uint) <br> } <br> MAD: struct CMFormat (256 bytes) - <br> Request for Communication (section 16.7.1) <br> { <br> MADHeader: struct MADHeader (24 bytes) - <br>
MAD Base Header (section 13.4.3) <br> { <br> baseVersion: 0x01 (8 bit uint) <br> mgmtClass: 0x07 (8 bit uint) <br> classVersion: 0x02 (8 bit uint) <br> method: 0x03 (8 bit uint) <br> status: 0x0000 (16 bit uint) <br>
classSpecific: 0x0000 (16 bit uint) <br> transactionID: 0x0000003BE64D1C3E (64 bit uint) <br> attributeID: 0x0010 (16 bit uint) <br> rsv0: 0x0000 (16 bit uint) <br> attributeModifier: 0x00000000 (32 bit uint) <br> } <br> data: struct CMREQ (232 bytes) - <br> Request for Communication (section 12.6.5) <br>
{ <br> LCID: 0x3E1C4DE6 (32 bit uint) <br> rsv0: 0x00000000 (32 bit uint) <br> serviceID: 0x2000020453000011 (64 bit uint) <br> LGUID: 0x00066A0098005B37 (64 bit uint) <br> localCMQKey: 0x00000000 (32 bit uint) <br> localQKey: 0x00000000 (32 bit
uint) <br> localQPN: 0x180014 (24 bit uint) <br> responderResources: 0x04 (8 bit uint) <br> localEECN: 0x000000 (24 bit uint) <br> initiatorDepth: 0x00 (8 bit uint) <br> remoteEECN: 0x000000 (24 bit uint) <br> remoteResponseTimeout: 0x14 (5 bit uint) <br> transportService: 0x0 (2 bit uint) <br> flowControl:
0x1 (1 bit uint) <br> startingPSN: 0x1D4319 (24 bit uint) <br> localResponseTimeout: 0x14 (5 bit uint) <br> retryCount: 0x7 (3 bit uint) <br> PKey: 0xFFFF (16 bit uint) <br> pathPacketMTU: 0x4 (4 bit uint) <br> RDCExists: 0x0 (1 bit uint) <br> RNRRetryCount: 0x7 (3 bit uint) <br>
maxCMRetries: 0xF (4 bit uint) <br> SRQ: 0x0 (1 bit uint) <br> rsv1: 0x0 (3 bit uint) <br> primaryPath: struct CMPath (44 bytes) - <br> Path Information (section 12.6) <br> { <br> SLID:
0x0104 (16 bit uint) <br> DLID: 0x0004 (16 bit uint) <br> SGID: FE80:0:0:0:6:6A00:A000:5B37 (HdrIPv6Addr) <br> DGID: FE80:0:0:0:4:5381:3000:30B2 (HdrIPv6Addr) <br> flowLabel: 0x00000 (20 bit uint) <br>
rsv0: 0x0 (4 bit uint) <br> rsv1: 0x0 (2 bit uint) <br> packetRate: 0x02 (6 bit uint) <br> TClass: 0x00 (8 bit uint) <br> hopLimit: 0x00 (8 bit uint) <br>
SL: 0x0 (4 bit uint) <br> subnetLocal: 0x1 (1 bit uint) <br> rsv2: 0x0 (3 bit uint) <br> localACKTimeout: 0x13 (5 bit uint) <br> rsv3: 0x0 (3 bit uint) <br> } <br> alternatePath: struct CMPath (44 bytes) - <br>
Path Information (section 12.6) <br> { <br> SLID: 0x0000 (16 bit uint) <br> DLID: 0x0000 (16 bit uint) <br> SGID: 0:0:0:0:0:0:0:0 (HdrIPv6Addr) <br> DGID:
0:0:0:0:0:0:0:0 (HdrIPv6Addr) <br> flowLabel: 0x00000 (20 bit uint) <br> rsv0: 0x0 (4 bit uint) <br> rsv1: 0x0 (2 bit uint) <br> packetRate: 0x00 (6 bit uint) <br> TClass: 0x00 (8 bit uint) <br>
hopLimit: 0x00 (8 bit uint) <br> SL: 0x0 (4 bit uint) <br> subnetLocal: 0x0 (1 bit uint) <br> rsv2: 0x0 (3 bit uint) <br> localACKTimeout: 0x00 (5 bit uint) <br>
rsv3: 0x0 (3 bit uint) <br> } <br> privateData: raw data in hex (92 bytes) <br> { <br> 00000000 00000000 00000000 00000000 00000104 00000000 00060000 <br> 00000000 00000000 00000000 00066A00 A0005B37 20000204 53000011 <br> 00045381 300030B2 00000000 00000000 00000000 00000000 00000000 <br> 00000000 00000000 <br> } <br> } <br> } <br> 0302070100000000 3b0000003e1c4de6 <br> 0000100000000000 e64d1c3e00000000 <br>
0402002011000053 006a0600375b0098 <br> 0000000000000000 0414001800000000 <br> a1000000a719431d f047ffff04000401 <br> 000080fe00000000 006a0600375b00a0 <br> 000080fe00000000 81530400b2300030 <br> 0200000098080000 0000000000000000 <br> 0000000000000000 0000000000000000 <br> 0000000000000000 0000000000000000 <br> 0000000000000000 0000000000000000 <br> 0000000004010000 0000000000000600 <br> 0000000000000000 00000000006a0600 <br> 375b00a004020020 1100005381530400 <br> b230003000000000 0000000000000000 <br> 0000000000000000 0000000000000000 <br> <br> -------------------------------------- <br> <br> *** CM REP *** <br> <br> sending MAD [QP1]: <br> MAD: struct CMFormat (256 bytes) - <br> Request for Communication (section 16.7.1) <br> { <br>
MADHeader: struct MADHeader (24 bytes) - <br> MAD Base Header (section 13.4.3) <br> { <br> baseVersion: 0x01 (8 bit uint) <br> mgmtClass: 0x07 (8 bit uint) <br> classVersion: 0x02 (8 bit uint) <br> method: 0x03 (8 bit uint) <br> status:
0x0000 (16 bit uint) <br> classSpecific: 0x0000 (16 bit uint) <br> transactionID: 0x0000003BE64D1C3E (64 bit uint) <br> attributeID: 0x0013 (16 bit uint) <br> rsv0: 0x0000 (16 bit uint) <br> attributeModifier: 0x00000000 (32 bit uint) <br> } <br> data: struct CMREP (232 bytes) - <br>
Reply To Request For Communication (section 12.6.8) <br> { <br> LCID: 0xE118A61B (32 bit uint) <br> RCID: 0x3E1C4DE6 (32 bit uint) <br> localQKey: 0x00000000 (32 bit uint) <br> localQPN: 0x000408 (24 bit uint) <br>
rsv0: 0x00 (8 bit uint) <br> localEEContext: 0x000000 (24 bit uint) <br> rsv1: 0x00 (8 bit uint) <br> startingPSN: 0x3B3163 (24 bit uint) <br> rsv2: 0x00 (8 bit uint) <br> responderResources: 0x04 (8 bit uint) <br> initiatorDepth: 0x07 (8 bit uint) <br>
targetACKDelay: 0x1F (5 bit uint) <br> failoverAccepted: 0x0 (2 bit uint) <br> flowControl: 0x1 (1 bit uint) <br> RNRRetryCount: 0x7 (3 bit uint) <br> rsv3: 0x00 (5 bit uint) <br> LGUID: 0x00045381300030B0 (64 bit uint) <br> privateData: raw data in hex (196 bytes) <br> { <br>
C0000000 00000064 00000000 00000000 00000104 00000104 00060000 <br> 00000000 00000000 00000000 00000000 00000000 00000000 00000000 <br> 00000000 00000000 00000000 00000000 00000000 00000000 00000000 <br> 00000000 00000000 00000000 00000000 00000000 00000000 00000000 <br> 00000000 00000000 00000000 00000000 00000000 00000000 00000000 <br> 00000000 00000000 00000000 00000000 00000000 00000000 00000000 <br> 00000000 00000000 00000000 00000000 00000000 00000000 00000000 <br> } <br> } <br> } <br>
0302070100000000 3b0000003e1c4de6 <br> 0000130000000000 1ba618e1e64d1c3e <br> 0000000000080400 000000000063313b <br> e0f9070481530400 b0300030000000c0 <br> 6400000000000000 0000000004010000 <br> 0401000000000600 0000000000000000 <br> 0000000000000000 0000000000000000 <br> 0000000000000000 0000000000000000 <br> 0000000000000000 0000000000000000 <br> 0000000000000000 0000000000000000 <br> 0000000000000000 0000000000000000 <br> 0000000000000000 0000000000000000 <br> 0000000000000000 0000000000000000 <br> 0000000000000000 0000000000000000 <br> 0000000000000000 0000000000000000 <br> 0000000000000000 0000000000000000 <br> -------------------------------------- <br> <br> *** CM REJ *** <br> <br> received MAD [QP1]: <br> struct HdrLRH (8 bytes) - <br> Local Route Header (section 7.7) <br> { <br>
VL: 0x0 (4 bit uint) <br> LVer: 0x0 (4 bit uint) <br> SL: 0x0 (4 bit uint) <br> rsv0: 0x0 (2 bit uint) <br> LNH: 0x2 (2 bit uint) <br> DLID: 0x0004 (16 bit uint) <br> rsv1:
0x00 (5 bit uint) <br> pktLen: 0x048 (11 bit uint) <br> SLID: 0x0104 (16 bit uint) <br> } <br> MAD: struct CMFormat (256 bytes) - <br> Request for Communication (section 16.7.1) <br> { <br> MADHeader: struct MADHeader (24 bytes) - <br> MAD Base Header (section 13.4.3) <br> { <br>
baseVersion: 0x01 (8 bit uint) <br> mgmtClass: 0x07 (8 bit uint) <br> classVersion: 0x02 (8 bit uint) <br> method: 0x03 (8 bit uint) <br> status: 0x0000 (16 bit uint) <br> classSpecific: 0x0000 (16 bit uint) <br> transactionID: 0x0000003BE64D1C3E (64 bit uint) <br>
attributeID: 0x0012 (16 bit uint) <br> rsv0: 0x0000 (16 bit uint) <br> attributeModifier: 0x00000000 (32 bit uint) <br> } <br> data: struct CMREJ (232 bytes) - <br> Reject (section 12.6.7) <br> { <br> LCID: 0x3E1C4DE6 (32 bit uint) <br>
RCID: 0xE118A61B (32 bit uint) <br> messageRejected: 0x1 (2 bit uint) <br> rsv0: 0x00 (6 bit uint) <br> rejectInfoLength: 0x00 (7 bit uint) <br> rsv1: 0x0 (1 bit uint) <br> reason: 0x001C (16 bit uint) <br> ARI:
raw data in hex (72 bytes) <br> { <br> 00000000 00000000 00000000 00000000 00000000 00000000 00000000 <br> 00000000 00000000 00000000 00000000 00000000 00000000 00000000 <br> 00000000 00000000 00000000 00000000 <br> } <br> privateData: raw data in hex (148 bytes) <br> { <br> 00000000 00000000 00000000 00000000 00000000 00000000 00000000 <br> 00000000 00000000 00000000 00000000 00000000 00000000 00000000 <br>
00000000 00000000 00000000 00000000 00000000 00000000 00000000 <br> 00000000 00000000 00000000 00000000 00000000 00000000 00000000 <br> 00000000 00000000 00000000 00000000 00000000 00000000 00000000 <br> 00000000 00000000 <br> } <br> } <br> } <br> 0302070100000000 3b0000003e1c4de6 <br> 0000120000000000 e64d1c3e1ba618e1 <br> 1c00004000000000 0000000000000000 <br> 0000000000000000 0000000000000000 <br> 0000000000000000 0000000000000000 <br> 0000000000000000 0000000000000000 <br> 0000000000000000 0000000000000000 <br> 0000000000000000 0000000000000000 <br> 0000000000000000 0000000000000000 <br> 0000000000000000 0000000000000000 <br> 0000000000000000 0000000000000000 <br> 0000000000000000
0000000000000000 <br> 0000000000000000 0000000000000000 <br> 0000000000000000 0000000000000000 <br> 0000000000000000 0000000000000000 <br> 0000000000000000 0000000000000000 <br> <br> -------------------------------------- <br> <br> <p>
<hr size=1>Make free worldwide PC-to-PC calls. Try the new <a href="http://ca.messenger.yahoo.com/"><b>Yahoo! Canada Messenger with Voice</b></a>