[ofa-general] Re: IB and iWarp HCA in same node

[Scott A. Friedman]

> Scott A. Friedman wrote:
>> > I have a working IB cluster where I have added a Chelsio iWarp card to 
>> > one node. Another node is connected to that with only an identical iWarp 
>> > card. I cannot seem to get the iWarp cards to come up. They work through 
>> > regular ethernet just fineand the IB stuff still works as well. But, 
>> > when I modprobe iw_cxgb3 and iw_cm utilities like ibstat show the 
>> > following. Which explains why nothing is working.
>> > 
>> > Question is, why? Am I missing or forgetting something? I just want to 
>> > test the two iWarp cards back to back. Not trying to get some kind of 
>> > auto bridging or routing working.
>> > 
>> > # ibstat
>> > iWARP RNIC 'cxgb3_0'
>> >         iWARP RNIC type: cxgb3
>> >         Number of ports: 1
>> >         Firmware version: T 4.0.0
>> >         Hardware version: 1
>> >         Node GUID: 0x0007430506ea0000
>> >         System image GUID: 0x0007430506ea0000
>> >         Port 1:
>> >                 State: Active
>> >                 Physical state: No state change
>> >                 Rate: 20
>> >                 Base lid: 0
>> >                 LMC: 0
>> >                 SM lid: 0
>> >                 Capability mask: 0x009f0000
>> >                 Port GUID: 0x0000000000000000
> 
> This all looks normal.  What application are you trying to run over rdma 
> on the chelsio interface?  rping?
> 

Yes, rping, anything. It turns out that since I posted this the Chelsio 
people explained ibstat's funny output and suggested using their latest 
release of the cxgb3 driver - and that works (without TOE for now, 
separate issue). The main problem was that the driver that ships with 
OFED would give me 'connection rejected' errors when trying to do 
anything (rdma_cm based), my code, sample code, utilities. Replacing the 
driver made the problem go away. Currently, I am using their 1.0.094 
driver w/o TOE and the OFED-1.2-rc3 iWarp stuff (their suggestion) and 
it appears to work fine so far. Going to just wait for rc5 or final to 
test that with their driver as well as that is what we will want to use 
for the rest of our test cluster using IB.