[Users] FCA Testing

[Susan Coulter]

On Aug 21, 2012, at 9:10 AM, Florent Parent wrote:

> Not tested here. I actually found this existed while at the OFED user day.
> 
> I'm glad you are trying this out. This would be interesting to us too. :)
> 
> We have a user code here that performs (communication wise) about 10
> fold better on a Qlogic based cluster over our Mellanox IB system. We
> have more tests to do, but we boiled this down to the connection rate
> (user job does MPI send/recv across multiple ranks) being better over
> the Qlogic system.
> 
> The Mellanox FCA or MXM would be interesting to tests to see if things
> improve. Although we have the (old) ConnectX (not the 2 or 3)

The documentation states ConnectX-2 is required for FCA, not sure about MXM.

If I can get some results from FCA, MXM is my next area of research.
I believe PSM really helps QLogic performance.
Many of our codes are running faster on our QLogic cluster.

I will keep the list updated with any significant results.

> generation, I would need to validate if this is compatible.
> 
> Florent
> 
> On Mon, Aug 20, 2012 at 6:49 PM, Susan Coulter <markus at lanl.gov> wrote:
>> Has anyone tested the collectives offload provided by MLNX OFED?
>> 
>> I have everything set up correctly according to the FCA documentation and
>> fca_managerd is running.
>> My attempts to run with "coll_fca_enable 1" are failing.
>> First with an inability to talk to umad port
>> 
>> ibwarn: [2337] mad_rpc_open_port: can't open UMAD port (mlx4_0:1)
>> 1345251776.473792 [FCA_DEV cja001 2337] dev.c:560 error Failed to initialize
>> SA: Cannot assign requested address
>> 
>> The permissions on /dev/infiniband/umad0 were opened up - which got rid of
>> the error messages.
>> Now it is dying and spitting out a bunch of empty core files - but nothing
>> in the job run log.
>> It looks like it is going to start, then poof !
>> ====================================
>> 
>> Susan Coulter
>> HPC-3 Network/Infrastructure
>> 505-667-8425
>> Increase the Peace...
>> An eye for an eye leaves the whole world blind
>> ====================================
>> 
>> 
>> _______________________________________________
>> Users mailing list
>> Users at lists.openfabrics.org
>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/users
>> 

====================================

Susan Coulter
HPC-3 Network/Infrastructure
505-667-8425
Increase the Peace...
An eye for an eye leaves the whole world blind
====================================

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/users/attachments/20120821/f5022f83/attachment.html>