[openib-general] Connectivity

[Sean Hubbell]

Hal Rosenstock wrote:

>On Fri, 2005-09-02 at 08:28, Sean Hubbell wrote:
>  
>
>>Again, thanks Hal. Yes, I can perform an ibping on all of the nodes so 
>>connectivity and the ports inbetween are up. I am almost positive now 
>>that this has something to do with the IPoIB. I am going to try to ping 
>>each node and then look at the arp table. Do you know of anything I can 
>>do to look specifically at the IPoIB "exchange".
>>    
>>
>
>If ibping works, UD unicast is working (and would work for that part of
>IPoIB). What I suspect is not working is multicast. I suspect some issue
>with the IPoIB broadcast group. So can you comment on the topology and
>provide an OpenSM log when run with verbose ? [Also can you down and
>then up all the ib<n> interfaces and see if connectivity is restored.
>Also, is the SM running on a node also running IPoIB ?]
>
>If not, you can debug this using the following:
>
>  
>
The "currently" topology of the system is 4 Dell PowerEdge 2.8 GHz 
machines with hyperthreading. There also is another DELL and then one 
day there will be 48 other nodes that are blades in 4 other chasises. 
There are 12 infiniband switches which basically use three switches to 
route to the other switches.

The log file I cannot send. I can go through it and answer any questions 
that you have. I realize this is stupid, but this is well above me.

I am not sure about the Subnet Manager. How can I tell where it is running?

>1. Using ibroute, you can display the multicast tables in the switches.
>Using ibtracert you can trace the route of a multicast group.
>
>        Multicast examples:
>                ibroute -M 4    # dump all non empty mlids of switch with lid 4
>                ibroute -M 4 0xc010 0xc020      # same, but with range
>                ibroute -M -n 4 # simple dump format
>
>        Multicast example:
>                ibtracert -m 0xc000 4 16        # show multicast path of mlid 0xc000 between lids 4 and 16
>
>  
>
I will try these.

>2. There are 2 levels of debug tracing in IPoIB. You can enable these in
>the build with CONFIG_INFINIBAND_IPOIB_DEBUG and
>CONFIG_INFINIBAND_IPOIB_DEBUG_DATA
>  
>
Sorry for my ignorance, but how would one go about doing this?

Sean