[ofa-general] poll CQ failed -2 with connectX

[Rick Warner]

Hi all,

I am configuring an opteron cluster with connectX Infiniband.  I have a 
problem that if I run one of the NAS tests, it works the first, and maybe 2nd 
time, but after that the jobs instantly fail with messages like this-

[Rank 44][cm.c: line 860]poll CQ failed -2
[Rank 51][cm.c: line 860]poll CQ failed -2
[Rank 119][cm.c: line 860]poll CQ failed -2
[Rank 85][cm.c: line 860]poll CQ failed -2
[Rank 0][cm.c: line 860]poll CQ failed -2
[Rank 9][cm.c: line 860]poll CQ failed -2
[Rank 26][cm.c: line 860]poll CQ failed -2[Rank 43][cm.c: line 860]
poll CQ failed -2
[Rank 94][cm.c: line 860]poll CQ failed -2
[Rank 111][cm.c: line 860]poll CQ failed -2

I can easily reproduce this with only 2 systems using a 16 process LU job, 
class B.

Here are the configs I've tried-
Suse 11 with distro provided IB driver and libraries,etc, using mvapich as 
provided by ohio state
Suse 11 with distro driver, using OFED 1.3.1 libraries and mvapich
Suse 10.3 with OFED 1.3.1, OFED 1.2.5.4, and OFED 1.4rc3

They all have the same basic problem.  I think one of them reported "Error 
polling CQ" instead of "poll CQ failed".

If I replace the connectX cards with regular DDR cards the problem goes away.

I'm getting quite stumped at this point and would appreciate any suggestions 
or patches.

Thanks,
Rick
-- 
Richard Warner
Lead Systems Integrator
Microway, Inc
(508)732-5517