[ofa-general] minimum sw components requirement for driver/opensm in a single unmanaged switch network

Wed Aug 27 15:56:52 PDT 2008

Yicheng Jia wrote:
> My operation is quite simple: connect QPs and do RDMA read/write. In this 
> case, the opensm is not in need when the subnet is up, correct?
>   
Is this a production subnet ? Do you need to deal with any failures ?

-- Hal

> Thanks!
> Yicheng
>
>
>
>
> "Dotan Barak" <dotanba at gmail.com> 
> 08/21/2008 02:33 PM
>
> To
> "Yicheng Jia" <YJia at tmriusa.com>
> cc
> "Hal Rosenstock" <hal.rosenstock at gmail.com>, general at lists.openfabrics.org
> Subject
> Re: [ofa-general] minimum sw components requirement for driver/opensm in a 
> single unmanaged switch network
>
>
>
>
>
>
> On Thu, Aug 21, 2008 at 10:16 PM, Yicheng Jia <YJia at tmriusa.com> wrote:
>   
>> Hi Hal,
>>
>> Can opensm just run once? When the subnet is up, it can exit assume that 
>>     
> no
>   
>> change will be made in the subnet.
>>
>>     
> Yes, depend on the serives that you will need/use.
>
> For example: if you use operations that requires SA query, you must
> have a live SM.
>
> If you will connect the QPs in the subnet by yourself (for example,
> using socket) you can manage without a live SM in the subnet ...
>
> Dotan
>   
>> Thanks!
>> Yicheng
>>
>>
>>
>> "Hal Rosenstock" <hal.rosenstock at gmail.com>
>>
>> 07/10/2008 09:15 PM
>>
>> To
>> "Yicheng Jia" <YJia at tmriusa.com>
>> cc
>> "Jim Mott" <jim at mellanox.com>, general at lists.openfabrics.org
>> Subject
>> Re: [ofa-general] minimum sw components requirement for driver/opensm in 
>>     
> a
>   
>> single unmanaged switch network
>>
>>
>>
>>
>> On Thu, Jul 10, 2008 at 7:39 PM, Yicheng Jia <YJia at tmriusa.com> wrote:
>>     
>>>> If you want to avoid all the SM stuff, and are willing to program the
>>>> switches directly (a few mads)
>>>>         
>>> Is it done by opensm?
>>>       
>> Yes.
>>
>>     
>>> What information should be set up in the switch by
>>> opensm?
>>>       
>> Things like the PortInfos and LFT. See IBA spec vol 1 14.2.5
>>
>>     
>>>> Then to figure out QP connections, you just use a function of 3
>>>> parameters:
>>>>  my_qp_num = fn_sqp(my_node, target_node, qp_num)
>>>>  target_qp_num = fn_tqp(my_node, target_node, qp_num)
>>>> Where qp_num is a small number between 0 and the maximum number of QPs
>>>> you
>>>> need active between any 2 endpoints.
>>>>         
>>> Can the qp_num be manually assigned?
>>> Does it need opensm be involved?
>>>       
>> SM has nothing to do with QP numbers.
>>
>>     
>>>> If it works, you are done.  If not, reset, up, wait for him to connect
>>>> and
>>>> send something to you.
>>>>         
>>> Is it reliable? I mean the QPs connection will keep alive during the 
>>>       
> QPs
>   
>>> lifecycle?
>>>       
>> For one thing, SM needs to try to keep ports at active.
>>
>> -- Hal
>>
>>     
>>> Best,
>>> Yicheng
>>>
>>>
>>>
>>> "Jim Mott" <jim at mellanox.com>
>>>
>>> 07/10/2008 04:17 PM
>>>
>>> To
>>> "Yicheng Jia" <YJia at tmriusa.com>, <general at lists.openfabrics.org>
>>> cc
>>> Subject
>>> RE: [ofa-general] minimum sw components requirement for driver/opensm 
>>>       
> in a
>   
>>> single unmanaged switch network
>>>
>>>
>>>
>>>
>>> If you want to avoid all the SM stuff, and are willing to program the
>>> switches directly (a few mads), then I've used schemes like:
>>>
>>> Node LID=base + (switch port * constant) (base=0, constant = 1 works)
>>>
>>> Then to figure out QP connections, you just use a function of 3
>>> parameters:
>>>   my_qp_num = fn_sqp(my_node, target_node, qp_num)
>>>   target_qp_num = fn_tqp(my_node, target_node, qp_num)
>>> Where qp_num is a small number between 0 and the maximum number of QPs 
>>>       
> you
>   
>>> need active between any 2 endpoints.
>>>
>>> With the above scheme, you know your node_id (switch port number), your
>>> lid,
>>> the lid of the target node,  and the QPs on both sides.  From there on, 
>>>       
> it
>   
>>> is clear sailing.  You don't even need to send MADs; just transition 
>>>       
> the
>   
>>> QP
>>> up and try and use it.  If it works, you are done.  If not, reset, up,
>>> wait
>>> for him to connect and send something to you.  A little timer to make 
>>>       
> sure
>   
>>> everybody retries once in awhile and what can go wrong?
>>>
>>> Jim
>>> From: general-bounces at lists.openfabrics.org
>>> [mailto:general-bounces at lists.openfabrics.org] On Behalf Of Yicheng Jia
>>> Sent: Thursday, July 10, 2008 2:59 PM
>>> To: general at lists.openfabrics.org
>>> Subject: [ofa-general] minimum sw components requirement for 
>>>       
> driver/opensm
>   
>>> in a single unmanaged switch network
>>>
>>>
>>> Hi Folks,
>>>
>>> I have a IB network which consists of only a single unmanaged switch, 
>>>       
> all
>   
>>> end nodes connecting with the switch only need to do RDMA read/write
>>> operation with each other. My question is, what are the indispensable
>>> modules in driver's core and opensm that make the network up and run?
>>>
>>> I've been using only ib_mad module in driver's core with a managed 
>>>       
> switch
>   
>>> before, and the network works fine. So I assume that only the ib_mad
>>> module
>>> in driver's core and SM in opensm are mandatory in my network. The LIDs
>>> are
>>> assigned by them. The SA and CM modules are not useful in my case. Am I
>>> right?
>>>
>>> I need to minimize driver and opensm to fit them in my network, the HCA
>>> driver is mthca.
>>>
>>> Best,
>>> Yicheng
>>>
>>>
>>>       
> _____________________________________________________________________________
>   
>>> Scanned by IBM Email Security Management Services powered by 
>>>       
> MessageLabs.
>   
>>> For more information please visit http://www.ers.ibm.com
>>>
>>>
>>>       
> _____________________________________________________________________________
>   
>>>
>>>       
> _____________________________________________________________________________
>   
>>> Scanned by IBM Email Security Management Services powered by 
>>>       
> MessageLabs.
>   
>>> For more information please visit http://www.ers.ibm.com
>>>
>>>
>>>       
> _____________________________________________________________________________
>   
>>>
>>>       
> _____________________________________________________________________________
>   
>>> Scanned by IBM Email Security Management Services powered by 
>>>       
> MessageLabs.
>   
>>> For more information please visit http://www.ers.ibm.com
>>>
>>>
>>>       
> _____________________________________________________________________________
>   
>>> _______________________________________________
>>> general mailing list
>>> general at lists.openfabrics.org
>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>>>
>>> To unsubscribe, please visit
>>> http://openib.org/mailman/listinfo/openib-general
>>>
>>>       
>>     
> _____________________________________________________________________________
>   
>> Scanned by IBM Email Security Management Services powered by 
>>     
> MessageLabs.
>   
>> For more information please visit http://www.ers.ibm.com
>>
>>     
> _____________________________________________________________________________
>   
>>
>>     
> _____________________________________________________________________________
>   
>> Scanned by IBM Email Security Management Services powered by 
>>     
> MessageLabs.
>   
>> For more information please visit http://www.ers.ibm.com
>>
>>     
> _____________________________________________________________________________
>   
>> _______________________________________________
>> general mailing list
>> general at lists.openfabrics.org
>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>>
>> To unsubscribe, please visit
>> http://openib.org/mailman/listinfo/openib-general
>>
>>     
>
> _____________________________________________________________________________
> Scanned by IBM Email Security Management Services powered by MessageLabs. 
> For more information please visit http://www.ers.ibm.com
> _____________________________________________________________________________
>
>
>
> _____________________________________________________________________________
> Scanned by IBM Email Security Management Services powered by MessageLabs. For more information please visit http://www.ers.ibm.com
> _____________________________________________________________________________
>   
> ------------------------------------------------------------------------
>
> _______________________________________________
> general mailing list
> general at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general