[ewg] RE: [ofa-general] RE: Agenda for the OFED meeting today (Jan 5, 09)

[John Russo]

Sorry Jeff... I am playing "middle man" with another engineer here for this information...

The query does not have to be per QP, but it does need to be per IB HCA port to IB HCA port communication path.

For example, if a node has 64 CPUs, it could do the query once per each other node on behalf of the 64 processes.

Its still an N^2 set of queries, but at least N can be reduced to be the number of end node IB ports as opposed to the number of processes.



-----Original Message-----
From: Jeff Squyres [mailto:jsquyres at cisco.com] 
Sent: Monday, January 05, 2009 12:07 PM
To: John Russo
Cc: Tziporet Koren; ewg at lists.openfabrics.org; general at lists.openfabrics.org
Subject: Re: [ofa-general] RE: Agenda for the OFED meeting today (Jan 5, 09)

Hmm.  Perhaps I'm not grokking your answer -- did you answer my  
question?  I'm indirectly asking about scalability of the SM to have  
hundreds/thousands of MPI processes simultaneously querying the SM.


On Jan 5, 2009, at 11:47 AM, John Russo wrote:

> When using routing algorithms such as lash, the SL used per end to  
> end connection will vary based on the route.  lash uses multiple VLs  
> to avoid credit loops.  As such the SL reported will vary based on  
> fabric topology and which pair of end nodes the path is being  
> requested on behalf of.
>
> -----Original Message-----
> From: Jeff Squyres [mailto:jsquyres at cisco.com]
> Sent: Monday, January 05, 2009 11:39 AM
> To: John Russo
> Cc: Tziporet Koren; ewg at lists.openfabrics.org; general at lists.openfabrics.org
> Subject: Re: [ofa-general] RE: Agenda for the OFED meeting today  
> (Jan 5, 09)
>
> Would all MPI processes need to query the SM for each path that they
> want to use in a QP?
>
>
> On Jan 5, 2009, at 11:31 AM, John Russo wrote:
>
>> Another suggestion for 1.5
>>
>> Implementation of SA queries for Path Records (using IBTA 1.2.1
>> ServiceId field) in all OFED ULPs, especially for MPI
>>            The IBTA standard defines that the proper way to
>> establish a connection is to get a PathRecord from the SM/SA and use
>> it to define all the attributes of the communication path.
>> Ideally the IBTA CM should then be used to establish the connection
>> and QPs as well.
>>
>> At present, openmpi, mvapich1 and mvapich2 do not use PathRecords,
>> but instead hard code attributes like the PKey, SL, etc.
>> In some cases these hardcoded values can be overridden by
>> configurable values such as PKey and SL, but such values must be
>> uniform across all connections and must be provided per job (which
>> can be error prone/tedious).
>>
>>            At present opensm supports PKeys and SLs, however MPI
>> cannot easily use these features.
>> Other features, such as lash routing, in opensm do not work properly
>> with MPI because the SL must be uniform across all connections, but
>> for lash it will vary per route.
>>
>> Additionally, applications which do not use PathRecords will have
>> difficulties with advanced features like IB routing, partitioning,
>> etc.  All of which are available or being worked on in opensm.
>>
>> From: ewg-bounces at lists.openfabrics.org [mailto:ewg-bounces at lists.openfabrics.org
>> ] On Behalf Of Tziporet Koren
>> Sent: Monday, January 05, 2009 1:00 AM
>> To: ewg at lists.openfabrics.org
>> Cc: general at lists.openfabrics.org
>> Subject: [ewg] Agenda for the OFED meeting today (Jan 5, 09)
>>
>>
>> Hello all,
>>
>> I hope we all had nice holidays and vacations, and now it's the time
>> to get back to business.
>>
>> Agenda for OFED meeting today:
>>
>> 1. Conclusions from OFED 1.4 release: Open discussion
>>
>> 2. Do we wish to have OFED 1.4.: Please send pros & cons before the
>> meeting
>>
>> 3. OFED 1.5: Schedule and features.
>>
>> This is what we presented in SC08 about 1.5:
>>
>> Preliminary Schedule:
>>
>> 	* Feature Freeze: 3/20/09
>> 	* Alpha Release: 3/20/09
>> 	* Beta Release: 4/20/09
>> 	* RC1: 5/5/09
>> 	* RC2-RCx: About every 2 weeks as needed
>> 	* Release: June 2009
>> Features:
>>
>> 	* Kernel.org: 2.6.28 and 2.6.29
>> 	* Multiple Event Queues to support Multi-core CPUs
>> 	* NFS/RDMA - GA
>> 	* RDS support for iWARP
>> 	* OpenMPI 1.3
>> 	* Add support/backports for RedHat EL 5.3 and EL 4.8, SLES 11
>> 	* Support for Mellanox vNIC (EoIB) and FCoIB with BridgeX device
>> 	* more TBD...
>>
>> We also presented the OS matrix but I suggest we will close this in
>> the next meeting.
>>
>> My proposal:
>>
>> 	* Have the release in July and not June - so we will have more time
>> for development
>> 	* Stick to one kernel version base and not change in the middle
>> since we saw that changing the kernel base caused a delay.
>> We need to decide in the meeting if it is 2.6.29 or we should wait
>> for 2.6.30.
>> 	* Add IB over Eth - this is similar to iWARP but more like IB (e.g.
>> including UD), and can work over ConnectX.
>>
>> Please send your suggestions to the list before the meeting if
>> possible
>>
>> Tziporet
>>
>>
>> _______________________________________________
>> general mailing list
>> general at lists.openfabrics.org
>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>>
>> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
>
>
> -- 
> Jeff Squyres
> Cisco Systems
>


-- 
Jeff Squyres
Cisco Systems