[ofw] OpenSM with HPC

[Yevgeny Kliteynik]

Hi Anatoly,

The OpenSM that you sent has many different SM executions, so I'm
not sure what execution should I refer to. However, I see a lot of
the following errors:

[23:28:03:918][152C] -> osm_vendor_send: ERR 3B28: Unable to create address vector (IB_INSUFFICIENT_MEMORY).
[23:28:03:918][152C] -> __osm_pr_rcv_respond: ERR 1F15: Unable to send MAD (IB_INSUFFICIENT_MEMORY)

Is there a problem with memory on the host?
This would explain why the host freezes.

I'm just trying to understand whether the problem is in OpenSM
(in which case, as we already mentioned, the right solution
would be updating/upgrading WinSM), or in the specific setup.

 > usage of OpenSM goes to ~30MB, cpu usage to 100% and eventually
 > the node freezes and needs to be reset.

30MB is peanuts. Who is eating all the memory?
Do these MPI jobs run from the SM node too?

Also, how do the MPI tasks communicate? What do they use to perform
all-to-all path resolution? IPoIB? Just a thought:
(32 Computing nodes, Quadcore Dualsocket) = 2^5 MPI tasks in each
MPI job, which is 2^10 path resolution queries.
256 concurrent jobs make it 2^16 path resolution queries (~65K),
so if they all try to do path resolution w/o some kind of queries
aggregation/caching, it will keep SM busy with 100% CPU for some time.

-- Yevgeny

Anatoly Greenblatt wrote:
> Hi,
> 
> Information I have so far:
> 
> 32 Computing nodes, Quadcore Dualsocket
> 1 Head node, Quadcore Dualsocket 64GB RAM
> 1 Control node, Dualcore Dualsocket
> 
> 3 edge switches
> 2 core switches
> 
> OpenSM running either on Head node or control node, in both cases the
> system freezes when running over 192 concurrent jobs. 
> 
> They need to run 256 MPI concurrent jobs continuously.
> 
> Regards,
> Anatoly.
> 
> 
> 
> -----Original Message-----
> From: Yevgeny Kliteynik [mailto:kliteyn at dev.mellanox.co.il] 
> Sent: Thursday, October 02, 2008 22:54
> To: Anatoly Greenblatt
> Cc: ofw at lists.openfabrics.org
> Subject: Re: [ofw] OpenSM with HPC
> 
> Hi Anatoly,
> 
> I need more details:
> 
> Anatoly Greenblatt wrote:
>> Hi,
>>
>> Our client reported problems running over 192 concurrent jobs with 
>> OpenSM.
> 
> What kind of cluster does your client have?
> How many hosts? How many switches?
> 
> What do these jobs do?
> Are these MPI jobs? Do they use/create multicast groups? Something else?
> How many processes each job has?
> 
>> The jobs are executed several times. After a while the memory 
>> usage of OpenSM goes to ~30MB, cpu usage to 100% and eventually the
> node 
>> freezes and needs to be reset.
> 
> Is the problem reproducible?
> Can you send me SM log?
> 
> -- Yevgeny
> 
>>  
>>
>> Configuration:
>>
>> Winof rev 1596 (~rc1)
>>
>> ConnectX HCA
>>
>> Windows 2008 x64 with HPC pack rc2
>>
>> NetworkDirect is installed
>>
>> OpenSM is running as a service on the head node.
>>
>> About a hundred nodes are used (maybe more, I don't have exact number
> yet)
>>  
>>
>> Has anyone any thoughts about this?
>>
>>  
>>
>> Thanks,
>>
>> Anatoly.
>>
>>  
>>
>>
>>
> ------------------------------------------------------------------------
>> _______________________________________________
>> ofw mailing list
>> ofw at lists.openfabrics.org
>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw
>