[openib-general] Multicast Group Routing Question

[Sean Hubbell]

Hal Rosenstock wrote:
> On Wed, 2006-12-06 at 16:06, Sean Hubbell wrote:
>   
>> Hal Rosenstock wrote:
>>     
>>> On Wed, 2006-12-06 at 13:48, Sean Hubbell wrote:
>>>   
>>>       
>>>> Hal Rosenstock wrote:
>>>>     
>>>>         
>>>>> Hi Sean,
>>>>>
>>>>> On Wed, 2006-12-06 at 12:52, Sean Hubbell wrote:
>>>>>   
>>>>>       
>>>>>           
>>>>>> Hello,
>>>>>>
>>>>>>   I was testing our code and noticed that when I send data using 
>>>>>> multicast over our ib0 interface, all of the infiniband switches route 
>>>>>> the data to each switch and each node instead of a node that has an 
>>>>>> application listening to the interface like Ethernet. Is this by design?
>>>>>>     
>>>>>>         
>>>>>>             
>>>>> It depends on what multicast group is being used and which end nodes
>>>>> have registered for that group as to where the data is routed.
>>>>>
>>>>> -- Hal
>>>>>   
>>>>>       
>>>>>           
>>>> Hey Hal,
>>>>
>>>>   The multicast group I am sending data to is 224.10.10.x (not 
>>>> 224.0.0.x) and I have no clients / nodes listening but the data is still 
>>>> being sent.
>>>>     
>>>>         
>>> Yes, if there is only a sender, the data should not be routed anywhere.
>>>
>>>   
>>>       
>>>>  I am using wwtop from warewulf to view the network load for 
>>>> each node.
>>>>     
>>>>         
>>> I'm not familiar with those tools.
>>>
>>>   
>>>       
>>>>  Does this make sense?
>>>>     
>>>>         
>>> Nope. To state the obvious, something is not as it seems...
>>>
>>> Can you state which SM you are using ?
>>>
>>> Also, can you do the following:
>>> saquery -g
>>> saquery -m
>>> and send me the output.
>>>
>>> I may have some more experiments once I get that level of info.
>>>
>>> -- Hal
>>>   
>>>       
>> We have a Voltaire HW subnet manager. I do not have the saquery command. 
>> I'll have to find this and install it.
>>     
>
> What is running on your end nodes ? Is it OpenIB/OFED or something else
> ? If it is OpenIB/OFED, saquery should be there. I think OFED 1.2
> supports the options I mentioned.
>
>   
>>  Would the web interface help?
>>     
>
> Not sure whether there is anything there for this.
>
> -- Hal
>
>   
>> Sean
>>     
>
>
>   
Hal,

  Here are the results:

The result of saquery -g on our head node:

[root at neptune ~]# saquery -g

MCMemberRecord group dump:
                                
MGID....................0xff12401bffff0000 : 0x00000000ffffffff
                                Mlid....................0xC000
                                Mtu.....................0x4
                                pkey....................0xFFFF
                                Rate....................0x3

MCMemberRecord group dump:
                                
MGID....................0xff12401bffff0000 : 0x0000000000000001
                                Mlid....................0xC001
                                Mtu.....................0x4
                                pkey....................0xFFFF
                                Rate....................0x3

The result of saquery -m on our root node:

Query SA failed: IB_TIMEOUT

Running package openib-diags-1.1.0-0

Sean