[ofa-general] minimum sw components requirement for driver/opensm in a single unmanaged switch network
Hal Rosenstock
halr at obsidianresearch.com
Thu Aug 28 04:32:48 PDT 2008
Yicheng Jia wrote:
> Yes. My basic idea is, the opensm set up the subnet during initialization,
> it will report errors during this process. After the subnet is up, the
> environment is fixed and stable. If some failure happens, opensm could be
> used again to diagnose the failure. From my understanding, in this case,
> the only work that opensm does after subnet is up is to log the status.
Wouldn't opensm also repair the failure if it could ?
-- Hal
>
> Thanks!
> Yicheng
>
>
>
>
> Hal Rosenstock <halr at obsidianresearch.com>
> 08/27/2008 05:55 PM
>
> To
> Yicheng Jia <YJia at tmriusa.com>
> cc
> Dotan Barak <dotanba at gmail.com>, general at lists.openfabrics.org
> Subject
> Re: [ofa-general] minimum sw components requirement for driver/opensm in a
> single unmanaged switch network
>
>
>
>
>
>
> Yicheng Jia wrote:
>
>> My operation is quite simple: connect QPs and do RDMA read/write. In
>>
> this
>
>> case, the opensm is not in need when the subnet is up, correct?
>>
>>
> Is this a production subnet ? Do you need to deal with any failures ?
>
> -- Hal
>
>
>> Thanks!
>> Yicheng
>>
>>
>>
>>
>> "Dotan Barak" <dotanba at gmail.com>
>> 08/21/2008 02:33 PM
>>
>> To
>> "Yicheng Jia" <YJia at tmriusa.com>
>> cc
>> "Hal Rosenstock" <hal.rosenstock at gmail.com>,
>>
> general at lists.openfabrics.org
>
>> Subject
>> Re: [ofa-general] minimum sw components requirement for driver/opensm in
>>
> a
>
>> single unmanaged switch network
>>
>>
>>
>>
>>
>>
>> On Thu, Aug 21, 2008 at 10:16 PM, Yicheng Jia <YJia at tmriusa.com> wrote:
>>
>>
>>> Hi Hal,
>>>
>>> Can opensm just run once? When the subnet is up, it can exit assume
>>>
> that
>
>> no
>>
>>
>>> change will be made in the subnet.
>>>
>>>
>>>
>> Yes, depend on the serives that you will need/use.
>>
>> For example: if you use operations that requires SA query, you must
>> have a live SM.
>>
>> If you will connect the QPs in the subnet by yourself (for example,
>> using socket) you can manage without a live SM in the subnet ...
>>
>> Dotan
>>
>>
>>> Thanks!
>>> Yicheng
>>>
>>>
>>>
>>> "Hal Rosenstock" <hal.rosenstock at gmail.com>
>>>
>>> 07/10/2008 09:15 PM
>>>
>>> To
>>> "Yicheng Jia" <YJia at tmriusa.com>
>>> cc
>>> "Jim Mott" <jim at mellanox.com>, general at lists.openfabrics.org
>>> Subject
>>> Re: [ofa-general] minimum sw components requirement for driver/opensm
>>>
> in
>
>> a
>>
>>
>>> single unmanaged switch network
>>>
>>>
>>>
>>>
>>> On Thu, Jul 10, 2008 at 7:39 PM, Yicheng Jia <YJia at tmriusa.com> wrote:
>>>
>>>
>>>>> If you want to avoid all the SM stuff, and are willing to program the
>>>>> switches directly (a few mads)
>>>>>
>>>>>
>>>> Is it done by opensm?
>>>>
>>>>
>>> Yes.
>>>
>>>
>>>
>>>> What information should be set up in the switch by
>>>> opensm?
>>>>
>>>>
>>> Things like the PortInfos and LFT. See IBA spec vol 1 14.2.5
>>>
>>>
>>>
>>>>> Then to figure out QP connections, you just use a function of 3
>>>>> parameters:
>>>>> my_qp_num = fn_sqp(my_node, target_node, qp_num)
>>>>> target_qp_num = fn_tqp(my_node, target_node, qp_num)
>>>>> Where qp_num is a small number between 0 and the maximum number of
>>>>>
> QPs
>
>>>>> you
>>>>> need active between any 2 endpoints.
>>>>>
>>>>>
>>>> Can the qp_num be manually assigned?
>>>> Does it need opensm be involved?
>>>>
>>>>
>>> SM has nothing to do with QP numbers.
>>>
>>>
>>>
>>>>> If it works, you are done. If not, reset, up, wait for him to
>>>>>
> connect
>
>>>>> and
>>>>> send something to you.
>>>>>
>>>>>
>>>> Is it reliable? I mean the QPs connection will keep alive during the
>>>>
>>>>
>> QPs
>>
>>
>>>> lifecycle?
>>>>
>>>>
>>> For one thing, SM needs to try to keep ports at active.
>>>
>>> -- Hal
>>>
>>>
>>>
>>>> Best,
>>>> Yicheng
>>>>
>>>>
>>>>
>>>> "Jim Mott" <jim at mellanox.com>
>>>>
>>>> 07/10/2008 04:17 PM
>>>>
>>>> To
>>>> "Yicheng Jia" <YJia at tmriusa.com>, <general at lists.openfabrics.org>
>>>> cc
>>>> Subject
>>>> RE: [ofa-general] minimum sw components requirement for driver/opensm
>>>>
>>>>
>> in a
>>
>>
>>>> single unmanaged switch network
>>>>
>>>>
>>>>
>>>>
>>>> If you want to avoid all the SM stuff, and are willing to program the
>>>> switches directly (a few mads), then I've used schemes like:
>>>>
>>>> Node LID=base + (switch port * constant) (base=0, constant = 1 works)
>>>>
>>>> Then to figure out QP connections, you just use a function of 3
>>>> parameters:
>>>> my_qp_num = fn_sqp(my_node, target_node, qp_num)
>>>> target_qp_num = fn_tqp(my_node, target_node, qp_num)
>>>> Where qp_num is a small number between 0 and the maximum number of QPs
>>>>
>
>
>> you
>>
>>
>>>> need active between any 2 endpoints.
>>>>
>>>> With the above scheme, you know your node_id (switch port number),
>>>>
> your
>
>>>> lid,
>>>> the lid of the target node, and the QPs on both sides. From there
>>>>
> on,
>
>> it
>>
>>
>>>> is clear sailing. You don't even need to send MADs; just transition
>>>>
>>>>
>> the
>>
>>
>>>> QP
>>>> up and try and use it. If it works, you are done. If not, reset, up,
>>>> wait
>>>> for him to connect and send something to you. A little timer to make
>>>>
>>>>
>> sure
>>
>>
>>>> everybody retries once in awhile and what can go wrong?
>>>>
>>>> Jim
>>>> From: general-bounces at lists.openfabrics.org
>>>> [mailto:general-bounces at lists.openfabrics.org] On Behalf Of Yicheng
>>>>
> Jia
>
>>>> Sent: Thursday, July 10, 2008 2:59 PM
>>>> To: general at lists.openfabrics.org
>>>> Subject: [ofa-general] minimum sw components requirement for
>>>>
>>>>
>> driver/opensm
>>
>>
>>>> in a single unmanaged switch network
>>>>
>>>>
>>>> Hi Folks,
>>>>
>>>> I have a IB network which consists of only a single unmanaged switch,
>>>>
>>>>
>> all
>>
>>
>>>> end nodes connecting with the switch only need to do RDMA read/write
>>>> operation with each other. My question is, what are the indispensable
>>>> modules in driver's core and opensm that make the network up and run?
>>>>
>>>> I've been using only ib_mad module in driver's core with a managed
>>>>
>>>>
>> switch
>>
>>
>>>> before, and the network works fine. So I assume that only the ib_mad
>>>> module
>>>> in driver's core and SM in opensm are mandatory in my network. The
>>>>
> LIDs
>
>>>> are
>>>> assigned by them. The SA and CM modules are not useful in my case. Am
>>>>
> I
>
>>>> right?
>>>>
>>>> I need to minimize driver and opensm to fit them in my network, the
>>>>
> HCA
>
>>>> driver is mthca.
>>>>
>>>> Best,
>>>> Yicheng
>>>>
>>>>
>>>>
>>>>
> _____________________________________________________________________________
>
>>>> Scanned by IBM Email Security Management Services powered by
>>>>
>>>>
>> MessageLabs.
>>
>>
>>>> For more information please visit http://www.ers.ibm.com
>>>>
>>>>
>>>>
>>>>
> _____________________________________________________________________________
>
>>>>
> _____________________________________________________________________________
>
>>>> Scanned by IBM Email Security Management Services powered by
>>>>
>>>>
>> MessageLabs.
>>
>>
>>>> For more information please visit http://www.ers.ibm.com
>>>>
>>>>
>>>>
>>>>
> _____________________________________________________________________________
>
>>>>
> _____________________________________________________________________________
>
>>>> Scanned by IBM Email Security Management Services powered by
>>>>
>>>>
>> MessageLabs.
>>
>>
>>>> For more information please visit http://www.ers.ibm.com
>>>>
>>>>
>>>>
>>>>
> _____________________________________________________________________________
>
>>>> _______________________________________________
>>>> general mailing list
>>>> general at lists.openfabrics.org
>>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>>>>
>>>> To unsubscribe, please visit
>>>> http://openib.org/mailman/listinfo/openib-general
>>>>
>>>>
>>>>
> _____________________________________________________________________________
>
>>> Scanned by IBM Email Security Management Services powered by
>>>
>>>
>> MessageLabs.
>>
>>
>>> For more information please visit http://www.ers.ibm.com
>>>
>>>
>>>
> _____________________________________________________________________________
>
>>>
> _____________________________________________________________________________
>
>>> Scanned by IBM Email Security Management Services powered by
>>>
>>>
>> MessageLabs.
>>
>>
>>> For more information please visit http://www.ers.ibm.com
>>>
>>>
>>>
> _____________________________________________________________________________
>
>>> _______________________________________________
>>> general mailing list
>>> general at lists.openfabrics.org
>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>>>
>>> To unsubscribe, please visit
>>> http://openib.org/mailman/listinfo/openib-general
>>>
>>>
>>>
>>
> _____________________________________________________________________________
>
>> Scanned by IBM Email Security Management Services powered by
>>
> MessageLabs.
>
>> For more information please visit http://www.ers.ibm.com
>>
>>
> _____________________________________________________________________________
>
>>
>>
>>
> _____________________________________________________________________________
>
>> Scanned by IBM Email Security Management Services powered by
>>
> MessageLabs. For more information please visit http://www.ers.ibm.com
>
> _____________________________________________________________________________
>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> general mailing list
>> general at lists.openfabrics.org
>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>>
>> To unsubscribe, please visit
>>
> http://openib.org/mailman/listinfo/openib-general
>
>
> _____________________________________________________________________________
> Scanned by IBM Email Security Management Services powered by MessageLabs.
> For more information please visit http://www.ers.ibm.com
> _____________________________________________________________________________
>
>
>
> _____________________________________________________________________________
> Scanned by IBM Email Security Management Services powered by MessageLabs. For more information please visit http://www.ers.ibm.com
> _____________________________________________________________________________
>
More information about the general
mailing list