[ofa-general] synchronize commands issued to MTHCA

[Roland Dreier]

 > I'm using OFED-1.0 and the problem I believe is related to command 
 > synchronization of HCA. The host issues a MAD_INF command at first and 
 > then a SW2HW_MTP command without waiting for the completion of the first 
 > command. Both of commands return with bad parameters error.

I guess you mean the MAD_IFC and SW2HW_MPT commands?  I've never heard
of a problem like that -- more details about your hardware/software
config and the exact symptoms you see would be helpful in debugging.

Anyway OFED 1.0 is ancient by now -- you are much better off just
using drivers from the standard kernel.  If you must use OFED, then
OFED 1.2 or even a 1.3 prerelease would be better.

 > My question is why there's no synchronization mechanism for the command 
 > execution on HCA, can I use "spin_lock" or "sem_wait" to synchronize 
 > between every command?

The HCA firmware allows multiple commands to be queued.  The
dev->cmd.event_sem semaphore is used to limit the number of
outstanding commands to the HCA's capabilities, and the
dev->cmd.hcr_mutex mutex is used to serialize the actual writing of
commands to the HCA.

There was a mmiowb() added to mthca_cmd_post() fairly recently that
might fix your problems if you are running on a large SGI Altix system.

 - R.