[ofa-general] minimum sw components requirement for driver/opensm in a single unmanaged switch network
Hal Rosenstock
halr at obsidianresearch.com
Thu Aug 28 12:30:53 PDT 2008
Yicheng Jia wrote:
> I missed it.
I'm not following what you mean by that.
> So it's better to keep opensm running all the time?
>
Yes, that's what the architecture says and it's for sound reasons.
-- Hal
> Thanks!
> Yicheng
>
>
>
> Hal Rosenstock <halr at obsidianresearch.com>
> 08/28/2008 06:31 AM
>
> To
> Yicheng Jia <YJia at tmriusa.com>
> cc
> Dotan Barak <dotanba at gmail.com>, general at lists.openfabrics.org
> Subject
> Re: [ofa-general] minimum sw components requirement for driver/opensm in a
> single unmanaged switch network
>
>
>
>
>
>
> Yicheng Jia wrote:
>
>> Yes. My basic idea is, the opensm set up the subnet during
>>
> initialization,
>
>> it will report errors during this process. After the subnet is up, the
>> environment is fixed and stable. If some failure happens, opensm could
>>
> be
>
>> used again to diagnose the failure. From my understanding, in this case,
>>
>
>
>> the only work that opensm does after subnet is up is to log the status.
>>
> Wouldn't opensm also repair the failure if it could ?
>
> -- Hal
>
>
>
>> Thanks!
>> Yicheng
>>
>>
>>
>>
>> Hal Rosenstock <halr at obsidianresearch.com>
>> 08/27/2008 05:55 PM
>>
>> To
>> Yicheng Jia <YJia at tmriusa.com>
>> cc
>> Dotan Barak <dotanba at gmail.com>, general at lists.openfabrics.org
>> Subject
>> Re: [ofa-general] minimum sw components requirement for driver/opensm in
>>
> a
>
>> single unmanaged switch network
>>
>>
>>
>>
>>
>>
>> Yicheng Jia wrote:
>>
>>
>>> My operation is quite simple: connect QPs and do RDMA read/write. In
>>>
>>>
>> this
>>
>>
>>> case, the opensm is not in need when the subnet is up, correct?
>>>
>>>
>>>
>> Is this a production subnet ? Do you need to deal with any failures ?
>>
>> -- Hal
>>
>>
>>
>>> Thanks!
>>> Yicheng
>>>
>>>
>>>
>>>
>>> "Dotan Barak" <dotanba at gmail.com>
>>> 08/21/2008 02:33 PM
>>>
>>> To
>>> "Yicheng Jia" <YJia at tmriusa.com>
>>> cc
>>> "Hal Rosenstock" <hal.rosenstock at gmail.com>,
>>>
>>>
>> general at lists.openfabrics.org
>>
>>
>>> Subject
>>> Re: [ofa-general] minimum sw components requirement for driver/opensm
>>>
> in
>
>> a
>>
>>
>>> single unmanaged switch network
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Thu, Aug 21, 2008 at 10:16 PM, Yicheng Jia <YJia at tmriusa.com> wrote:
>>>
>>>
>>>
>>>> Hi Hal,
>>>>
>>>> Can opensm just run once? When the subnet is up, it can exit assume
>>>>
>>>>
>> that
>>
>>
>>> no
>>>
>>>
>>>
>>>> change will be made in the subnet.
>>>>
>>>>
>>>>
>>>>
>>> Yes, depend on the serives that you will need/use.
>>>
>>> For example: if you use operations that requires SA query, you must
>>> have a live SM.
>>>
>>> If you will connect the QPs in the subnet by yourself (for example,
>>> using socket) you can manage without a live SM in the subnet ...
>>>
>>> Dotan
>>>
>>>
>>>
>>>> Thanks!
>>>> Yicheng
>>>>
>>>>
>>>>
>>>> "Hal Rosenstock" <hal.rosenstock at gmail.com>
>>>>
>>>> 07/10/2008 09:15 PM
>>>>
>>>> To
>>>> "Yicheng Jia" <YJia at tmriusa.com>
>>>> cc
>>>> "Jim Mott" <jim at mellanox.com>, general at lists.openfabrics.org
>>>> Subject
>>>> Re: [ofa-general] minimum sw components requirement for driver/opensm
>>>>
>>>>
>> in
>>
>>
>>> a
>>>
>>>
>>>
>>>> single unmanaged switch network
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Jul 10, 2008 at 7:39 PM, Yicheng Jia <YJia at tmriusa.com> wrote:
>>>>
>>>>
>>>>
>>>>>> If you want to avoid all the SM stuff, and are willing to program
>>>>>>
> the
>
>>>>>> switches directly (a few mads)
>>>>>>
>>>>>>
>>>>>>
>>>>> Is it done by opensm?
>>>>>
>>>>>
>>>>>
>>>> Yes.
>>>>
>>>>
>>>>
>>>>
>>>>> What information should be set up in the switch by
>>>>> opensm?
>>>>>
>>>>>
>>>>>
>>>> Things like the PortInfos and LFT. See IBA spec vol 1 14.2.5
>>>>
>>>>
>>>>
>>>>
>>>>>> Then to figure out QP connections, you just use a function of 3
>>>>>> parameters:
>>>>>> my_qp_num = fn_sqp(my_node, target_node, qp_num)
>>>>>> target_qp_num = fn_tqp(my_node, target_node, qp_num)
>>>>>> Where qp_num is a small number between 0 and the maximum number of
>>>>>>
>>>>>>
>> QPs
>>
>>
>>>>>> you
>>>>>> need active between any 2 endpoints.
>>>>>>
>>>>>>
>>>>>>
>>>>> Can the qp_num be manually assigned?
>>>>> Does it need opensm be involved?
>>>>>
>>>>>
>>>>>
>>>> SM has nothing to do with QP numbers.
>>>>
>>>>
>>>>
>>>>
>>>>>> If it works, you are done. If not, reset, up, wait for him to
>>>>>>
>>>>>>
>> connect
>>
>>
>>>>>> and
>>>>>> send something to you.
>>>>>>
>>>>>>
>>>>>>
>>>>> Is it reliable? I mean the QPs connection will keep alive during the
>>>>>
>>>>>
>>>>>
>>> QPs
>>>
>>>
>>>
>>>>> lifecycle?
>>>>>
>>>>>
>>>>>
>>>> For one thing, SM needs to try to keep ports at active.
>>>>
>>>> -- Hal
>>>>
>>>>
>>>>
>>>>
>>>>> Best,
>>>>> Yicheng
>>>>>
>>>>>
>>>>>
>>>>> "Jim Mott" <jim at mellanox.com>
>>>>>
>>>>> 07/10/2008 04:17 PM
>>>>>
>>>>> To
>>>>> "Yicheng Jia" <YJia at tmriusa.com>, <general at lists.openfabrics.org>
>>>>> cc
>>>>> Subject
>>>>> RE: [ofa-general] minimum sw components requirement for driver/opensm
>>>>>
>
>
>>>>>
>>> in a
>>>
>>>
>>>
>>>>> single unmanaged switch network
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> If you want to avoid all the SM stuff, and are willing to program the
>>>>> switches directly (a few mads), then I've used schemes like:
>>>>>
>>>>> Node LID=base + (switch port * constant) (base=0, constant = 1 works)
>>>>>
>>>>> Then to figure out QP connections, you just use a function of 3
>>>>> parameters:
>>>>> my_qp_num = fn_sqp(my_node, target_node, qp_num)
>>>>> target_qp_num = fn_tqp(my_node, target_node, qp_num)
>>>>> Where qp_num is a small number between 0 and the maximum number of
>>>>>
> QPs
>
>>
>>> you
>>>
>>>
>>>
>>>>> need active between any 2 endpoints.
>>>>>
>>>>> With the above scheme, you know your node_id (switch port number),
>>>>>
>>>>>
>> your
>>
>>
>>>>> lid,
>>>>> the lid of the target node, and the QPs on both sides. From there
>>>>>
>>>>>
>> on,
>>
>>
>>> it
>>>
>>>
>>>
>>>>> is clear sailing. You don't even need to send MADs; just transition
>>>>>
>>>>>
>>>>>
>>> the
>>>
>>>
>>>
>>>>> QP
>>>>> up and try and use it. If it works, you are done. If not, reset,
>>>>>
> up,
>
>>>>> wait
>>>>> for him to connect and send something to you. A little timer to make
>>>>>
>
>
>>>>>
>>> sure
>>>
>>>
>>>
>>>>> everybody retries once in awhile and what can go wrong?
>>>>>
>>>>> Jim
>>>>> From: general-bounces at lists.openfabrics.org
>>>>> [mailto:general-bounces at lists.openfabrics.org] On Behalf Of Yicheng
>>>>>
>>>>>
>> Jia
>>
>>
>>>>> Sent: Thursday, July 10, 2008 2:59 PM
>>>>> To: general at lists.openfabrics.org
>>>>> Subject: [ofa-general] minimum sw components requirement for
>>>>>
>>>>>
>>>>>
>>> driver/opensm
>>>
>>>
>>>
>>>>> in a single unmanaged switch network
>>>>>
>>>>>
>>>>> Hi Folks,
>>>>>
>>>>> I have a IB network which consists of only a single unmanaged switch,
>>>>>
>
>
>>>>>
>>> all
>>>
>>>
>>>
>>>>> end nodes connecting with the switch only need to do RDMA read/write
>>>>> operation with each other. My question is, what are the indispensable
>>>>> modules in driver's core and opensm that make the network up and run?
>>>>>
>>>>> I've been using only ib_mad module in driver's core with a managed
>>>>>
>>>>>
>>>>>
>>> switch
>>>
>>>
>>>
>>>>> before, and the network works fine. So I assume that only the ib_mad
>>>>> module
>>>>> in driver's core and SM in opensm are mandatory in my network. The
>>>>>
>>>>>
>> LIDs
>>
>>
>>>>> are
>>>>> assigned by them. The SA and CM modules are not useful in my case. Am
>>>>>
>
>
>> I
>>
>>
>>>>> right?
>>>>>
>>>>> I need to minimize driver and opensm to fit them in my network, the
>>>>>
>>>>>
>> HCA
>>
>>
>>>>> driver is mthca.
>>>>>
>>>>> Best,
>>>>> Yicheng
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
> _____________________________________________________________________________
>
>>>>> Scanned by IBM Email Security Management Services powered by
>>>>>
>>>>>
>>>>>
>>> MessageLabs.
>>>
>>>
>>>
>>>>> For more information please visit http://www.ers.ibm.com
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
> _____________________________________________________________________________
>
> _____________________________________________________________________________
>
>>>>> Scanned by IBM Email Security Management Services powered by
>>>>>
>>>>>
>>>>>
>>> MessageLabs.
>>>
>>>
>>>
>>>>> For more information please visit http://www.ers.ibm.com
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
> _____________________________________________________________________________
>
> _____________________________________________________________________________
>
>>>>> Scanned by IBM Email Security Management Services powered by
>>>>>
>>>>>
>>>>>
>>> MessageLabs.
>>>
>>>
>>>
>>>>> For more information please visit http://www.ers.ibm.com
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
> _____________________________________________________________________________
>
>>>>> _______________________________________________
>>>>> general mailing list
>>>>> general at lists.openfabrics.org
>>>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>>>>>
>>>>> To unsubscribe, please visit
>>>>> http://openib.org/mailman/listinfo/openib-general
>>>>>
>>>>>
>>>>>
>>>>>
> _____________________________________________________________________________
>
>>>> Scanned by IBM Email Security Management Services powered by
>>>>
>>>>
>>>>
>>> MessageLabs.
>>>
>>>
>>>
>>>> For more information please visit http://www.ers.ibm.com
>>>>
>>>>
>>>>
>>>>
> _____________________________________________________________________________
>
> _____________________________________________________________________________
>
>>>> Scanned by IBM Email Security Management Services powered by
>>>>
>>>>
>>>>
>>> MessageLabs.
>>>
>>>
>>>
>>>> For more information please visit http://www.ers.ibm.com
>>>>
>>>>
>>>>
>>>>
> _____________________________________________________________________________
>
>>>> _______________________________________________
>>>> general mailing list
>>>> general at lists.openfabrics.org
>>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>>>>
>>>> To unsubscribe, please visit
>>>> http://openib.org/mailman/listinfo/openib-general
>>>>
>>>>
>>>>
>>>>
> _____________________________________________________________________________
>
>>> Scanned by IBM Email Security Management Services powered by
>>>
>>>
>> MessageLabs.
>>
>>
>>> For more information please visit http://www.ers.ibm.com
>>>
>>>
>>>
> _____________________________________________________________________________
>
>>>
>>>
> _____________________________________________________________________________
>
>>> Scanned by IBM Email Security Management Services powered by
>>>
>>>
>> MessageLabs. For more information please visit http://www.ers.ibm.com
>>
>>
>>
> _____________________________________________________________________________
>
> ------------------------------------------------------------------------
>
>>> _______________________________________________
>>> general mailing list
>>> general at lists.openfabrics.org
>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>>>
>>> To unsubscribe, please visit
>>>
>>>
>> http://openib.org/mailman/listinfo/openib-general
>>
>>
>>
>>
> _____________________________________________________________________________
>
>> Scanned by IBM Email Security Management Services powered by
>>
> MessageLabs.
>
>> For more information please visit http://www.ers.ibm.com
>>
>>
> _____________________________________________________________________________
>
>>
>>
>>
> _____________________________________________________________________________
>
>> Scanned by IBM Email Security Management Services powered by
>>
> MessageLabs. For more information please visit http://www.ers.ibm.com
>
> _____________________________________________________________________________
>
>
>
> _____________________________________________________________________________
> Scanned by IBM Email Security Management Services powered by MessageLabs.
> For more information please visit http://www.ers.ibm.com
> _____________________________________________________________________________
>
>
>
> _____________________________________________________________________________
> Scanned by IBM Email Security Management Services powered by MessageLabs. For more information please visit http://www.ers.ibm.com
> _____________________________________________________________________________
>
More information about the general
mailing list