<br><font size=2 face="sans-serif">Hi Jack,</font>
<br>
<br><font size=2 face="sans-serif">Thanks for your reply. The HCA I'm using
is memory free, the chip is MT25204 and the HCA type is arbel, so it doesn't
go through the "</font><tt><font size=2>if (ah->type == MTHCA_AH_ON_HCA</font></tt><font size=2 face="sans-serif">)"
part of code. By checking the debug output, I got more details about this
problem:</font>
<br>
<br><font size=2 face="sans-serif">The </font><tt><font size=2>SW2HW_MPT
command</font></tt><font size=2 face="sans-serif"> is issued while UDAV
table is been creating. During the time that the driver is waiting for
the completion of the command, it does many other things: creating send
mad package, posting send mad request to the SQ and posting another receive
mad request to the RQ. There's no error report for all of these actions.
However after it, the HCA report command parameter error for the </font><tt><font size=2>SW2HW_MPT.</font></tt>
<br>
<br><font size=2 face="sans-serif">I've copied a snippet context of the
debug trace output when this error happens, hopefully it will help spot
the reason.</font>
<br>
<br><font size=2 face="sans-serif">139903841835 HCR CMD: op_code:
LE: d</font>
<br><font size=2 face="sans-serif">139903861104 TRACE: mad.c:639/ib_mad_recv_done_handler</font>
<br><font size=2 face="sans-serif">139903890876 HCR CMD: in_param_h:
LE: 0</font>
<br><font size=2 face="sans-serif">139903942869 TRACE: mad.c:644/ib_mad_recv_done_handler</font>
<br><font size=2 face="sans-serif">139903993296 HCR CMD: in_param_l:
LE: cf616000</font>
<br><font size=2 face="sans-serif">139904038413 TRACE: verbs.c:182/ib_create_ah_from_wc</font>
<br><font size=2 face="sans-serif">139904094753 HCR CMD: input_modifier:
LE: 1e</font>
<br><font size=2 face="sans-serif">139904139150 TRACE: mthca_provider.c:447/mthca_ah_create</font>
<br><font size=2 face="sans-serif">MTHCA DBG: <mthca_av.c:229> Created
UDAV at 8075220/00000000:</font>
<br><font size=2 face="sans-serif">139904197065 HCR CMD: out_pram_h:
LE: 0</font>
<br><font size=2 face="sans-serif">139904333343 [ 0] 01000005</font>
<br><font size=2 face="sans-serif">139904384499 HCR CMD: out_pram_l:
LE: 0</font>
<br><font size=2 face="sans-serif">139904428086 [ 4] 0000ffff</font>
<br><font size=2 face="sans-serif">139904478675 HCR CMD: token:
LE: ffff0000</font>
<br><font size=2 face="sans-serif">139904520156 [ 8] 00003000</font>
<br><font size=2 face="sans-serif">139904572059 HCR CMD: op_code_modifier:
LE: 0</font>
<br><font size=2 face="sans-serif">139904612802 [ c] 00000000</font>
<br><font size=2 face="sans-serif">139904667693 HCR CMD: event:
LE: 0</font>
<br><font size=2 face="sans-serif">139904708526 [10] 00000000</font>
<br><font size=2 face="sans-serif">139904758422 HCR CMD 0x18h:
LE=80000d, BE=d008000</font>
<br><font size=2 face="sans-serif">139904799210 [14] 00000000</font>
<br><font size=2 face="sans-serif">139904904204 [18] 00000000</font>
<br><font size=2 face="sans-serif">139904946792MTHCA DBG: <mthca_cmd.c:195>
HCR_STATUS 40100698= d008000 ? 8000</font>
<br><font size=2 face="sans-serif"> [1c] 00000002</font>
<br><font size=2 face="sans-serif">139905076860 TRACE: mthca_av.c:235/mthca_create_ah</font>
<br><font size=2 face="sans-serif">139905112329 TRACE: mthca_av.c:243/mthca_create_ah</font>
<br><font size=2 face="sans-serif">139905147672 TRACE: mthca_provider.c:460/mthca_ah_create</font>
<br><font size=2 face="sans-serif">636959 DEBUG: <mthca_qp.c:1908>
Start mthca_arbel_post_send. qp 0 wr 8d984b8 </font>
<br><font size=2 face="sans-serif">139905324432 TRACE: mthca_qp.c:1911/mthca_arbel_post_send</font>
<br><font size=2 face="sans-serif">139905359505 TRACE: mthca_qp.c:1939/mthca_arbel_post_send</font>
<br><font size=2 face="sans-serif">139905418932 TRACE: mthca_qp.c:1949/mthca_arbel_post_send</font>
<br><font size=2 face="sans-serif">636959 DEBUG: <mthca_qp.c:1953>
qp is not direct access and wqe: 0x8d84400 </font>
<br><font size=2 face="sans-serif">139905541467 TRACE: mthca_qp.c:1954/mthca_arbel_post_send</font>
<br><font size=2 face="sans-serif">139905577647 TRACE: mthca_qp.c:1964/mthca_arbel_post_send</font>
<br><font size=2 face="sans-serif">139905614565 TRACE: mthca_qp.c:2057/mthca_arbel_post_send</font>
<br><font size=2 face="sans-serif">139905669411 TRACE: mthca_qp.c:2076/mthca_arbel_post_send</font>
<br><font size=2 face="sans-serif">139905705726 TRACE: mthca_qp.c:2078/mthca_arbel_post_send</font>
<br><font size=2 face="sans-serif">636959 DEBUG: <mthca_qp.c:2087>
wr sg length 0x18, lkey 0x80001900, local addr 0xce2393b8</font>
<br><font size=2 face="sans-serif">139905831060 TRACE: mthca_qp.c:2078/mthca_arbel_post_send</font>
<br><font size=2 face="sans-serif">636959 DEBUG: <mthca_qp.c:2087>
wr sg length 0xe8, lkey 0x80001900, local addr 0xce2393d0</font>
<br><font size=2 face="sans-serif">139905956322 TRACE: mthca_qp.c:2092/mthca_arbel_post_send</font>
<br><font size=2 face="sans-serif">636959 DEBUG: <mthca_qp.c:2101>
wr id 148473016</font>
<br><font size=2 face="sans-serif">139906069875 TRACE: mthca_qp.c:2120/mthca_arbel_post_send</font>
<br><font size=2 face="sans-serif">139906106379 TRACE: mthca_qp.c:2128/mthca_arbel_post_send</font>
<br><font size=2 face="sans-serif">139906142892 TRACE: mthca_qp.c:2131/mthca_arbel_post_send</font>
<br><font size=2 face="sans-serif">139906178640 TRACE: mthca_qp.c:2135/mthca_arbel_post_send</font>
<br><font size=2 face="sans-serif">139906214703 TRACE: mthca_qp.c:2158/mthca_arbel_post_send</font>
<br><font size=2 face="sans-serif">139906250568 TRACE: mthca_qp.c:2160/mthca_arbel_post_send</font>
<br><font size=2 face="sans-serif">636959 DEBUG: <mthca_qp.c:2162>
End mthca_arbel_post_send. err 0</font>
<br><font size=2 face="sans-serif"> 139906369953 TRACE: mad.c:650/ib_mad_recv_done_handler</font>
<br><font size=2 face="sans-serif">139906406295 TRACE: mad.c:669/ib_mad_recv_done_handler</font>
<br><font size=2 face="sans-serif">139906441539 TRACE: mad.c:672/ib_mad_recv_done_handler</font>
<br><font size=2 face="sans-serif">636959 QNX DBG: <mad.c:530>
mad_priv->header.mad_list.mad_queue->list.prev 88b0a2c </font>
<br><font size=2 face="sans-serif">139906578384 TRACE: mthca_qp.c:2177/mthca_arbel_post_receive</font>
<br><font size=2 face="sans-serif">139906614168 TRACE: mthca_qp.c:2194/mthca_arbel_post_receive</font>
<br><font size=2 face="sans-serif">139906649295 TRACE: mthca_qp.c:2196/mthca_arbel_post_receive</font>
<br><font size=2 face="sans-serif">139906689129 TRACE: mad.c:674/ib_mad_recv_done_handler</font>
<br><font size=2 face="sans-serif">139906723068 TRACE: mad.c:676/ib_mad_recv_done_handler</font>
<br><font size=2 face="sans-serif">636959 QNX DBG: <linux_cache.c:151>
kmem_cache 5 free object=88b0724</font>
<br><font size=2 face="sans-serif">139906793007 HCR CMD: Status Return:
: 3</font>
<br>
<br><font size=2 face="sans-serif">Again, thanks for your help!</font>
<br>
<br><font size=2 face="sans-serif">Best,</font>
<br><font size=2 face="sans-serif">Yicheng</font>
<br>
<br>
<br>
<br>
<table width=100%>
<tr valign=top>
<td width=40%><font size=1 face="sans-serif"><b>Jack Morgenstein <jackm@dev.mellanox.co.il></b>
</font>
<p><font size=1 face="sans-serif">01/01/2008 01:03 AM</font>
<td width=59%>
<table width=100%>
<tr valign=top>
<td>
<div align=right><font size=1 face="sans-serif">To</font></div>
<td><font size=1 face="sans-serif">general@lists.openfabrics.org</font>
<tr valign=top>
<td>
<div align=right><font size=1 face="sans-serif">cc</font></div>
<td><font size=1 face="sans-serif">Yicheng Jia <YJia@tmriusa.com>,
Roland Dreier <rdreier@cisco.com></font>
<tr valign=top>
<td>
<div align=right><font size=1 face="sans-serif">Subject</font></div>
<td><font size=1 face="sans-serif">Re: [ofa-general] synchronize commands
issued to MTHCA</font></table>
<br>
<table>
<tr valign=top>
<td>
<td></table>
<br></table>
<br>
<br>
<br><tt><font size=2>On Tuesday 01 January 2008 03:02, Yicheng Jia wrote:<br>
<br>
Does your HCA use on-board memory?<br>
(Run: "lspci" and look at "Mellanox" lines. You
have on-board memory<br>
if you see either:<br>
PCI bridge: Mellanox Technologies MT23108 InfiniHost HCA bridge (rev a1)<br>
InfiniBand: Mellanox Technologies MT23108 InfiniHost HCA (rev a1)<br>
OR:<br>
InfiniBand: Mellanox Technologies MT25208 InfiniHost III Ex (Tavor
compatibility mode)<br>
)<br>
<br>
In that case, when you create an AH in kernel space<br>
(file mthca_av.c, procedure mthca_create_ah() ), you will enter the following
flow:<br>
if (ah->type == MTHCA_AH_ON_HCA) {<br>
memcpy_toio(dev->av_table.av_map
+ index * MTHCA_AV_SIZE,<br>
av, MTHCA_AV_SIZE);<br>
kfree(av);<br>
}<br>
<br>
Roland, do you think that the memcpy_toio() call might mess things up?<br>
<br>
Maybe we need "wmb()" or "mmiowb()" here as well?<br>
<br>
- Jack<br>
<br>
> Hi Roland,<br>
> <br>
> Thanks for your reply!<br>
> <br>
> Actually I'm working on porting IB driver to QNX platform. I resume
the <br>
> work started by my former colleague, and I just found that the sync
codes <br>
> (dev->cmd.poll_sem and dev->cmd.hcr_mutex) were deleted for
unknown <br>
> reason. After adding back these sync codes, the driver runs much <br>
> smoothlier. <br>
> <br>
> However I still get a command exec error which I believe is relevant
to <br>
> command synchronization. The problem is when "Created UDAV"
is called <br>
> during SW2HW_MPT command is being executed, the SW2HW_MPT command
would <br>
> return with bad parameter error. Here are my debug trace output:<br>
> <br>
> 139903841835 HCR CMD: op_code: LE:
d<br>
> 139903861104 TRACE: mad.c:639/ib_mad_recv_done_handler<br>
> 139903890876 HCR CMD: in_param_h: LE: 0<br>
> 139903942869 TRACE: mad.c:644/ib_mad_recv_done_handler<br>
> 139903993296 HCR CMD: in_param_l: LE: cf616000<br>
> 139904038413 TRACE: verbs.c:182/ib_create_ah_from_wc<br>
> 139904094753 HCR CMD: input_modifier: LE: 1e<br>
> 139904139150 TRACE: mthca_provider.c:447/mthca_ah_create<br>
> MTHCA DBG: <mthca_av.c:229> Created UDAV at 8075220/00000000:<br>
> 139904197065 HCR CMD: out_pram_h: LE: 0<br>
> 139904333343 [ 0] 01000005<br>
> 139904384499 HCR CMD: out_pram_l: LE: 0<br>
> 139904428086 [ 4] 0000ffff<br>
> 139904478675 HCR CMD: token: LE:
ffff0000<br>
> 139904520156 [ 8] 00003000<br>
> 139904572059 HCR CMD: op_code_modifier: LE: 0<br>
> 139904612802 [ c] 00000000<br>
> 139904667693 HCR CMD: event: LE:
0<br>
> 139904708526 [10] 00000000<br>
> 139904758422 HCR CMD 0x18h:
LE=80000d, BE=d008000<br>
> 139904799210 [14] 00000000<br>
> 139904904204 [18] 00000000<br>
> 139904946792MTHCA DBG: <mthca_cmd.c:195> HCR_STATUS 40100698=
d008000 ? <br>
> 8000<br>
> [1c] 00000002<br>
> 139905076860 TRACE: mthca_av.c:235/mthca_create_ah<br>
> 139905112329 TRACE: mthca_av.c:243/mthca_create_ah<br>
> 139905147672 TRACE: mthca_provider.c:460/mthca_ah_create<br>
> ....<br>
> 139906793007 HCR CMD: Status Return:
: 3<br>
> <br>
> Do you have any idea?<br>
> <br>
> Thanks and have a good new year!<br>
> Yicheng<br>
> <br>
> <br>
> <br>
> <br>
> Roland Dreier <rdreier@cisco.com> <br>
> 12/28/2007 11:39 PM<br>
> <br>
> To<br>
> Yicheng Jia <YJia@tmriusa.com><br>
> cc<br>
> general@lists.openfabrics.org<br>
> Subject<br>
> Re: [ofa-general] synchronize commands issued to MTHCA<br>
> <br>
> <br>
> <br>
> <br>
> <br>
> <br>
> > I'm using OFED-1.0 and the problem I believe is related
to command <br>
> > synchronization of HCA. The host issues a MAD_INF command
at first and <br>
> > then a SW2HW_MTP command without waiting for the completion
of the <br>
> first <br>
> > command. Both of commands return with bad parameters error.<br>
> <br>
> I guess you mean the MAD_IFC and SW2HW_MPT commands? I've never
heard<br>
> of a problem like that -- more details about your hardware/software<br>
> config and the exact symptoms you see would be helpful in debugging.<br>
> <br>
> Anyway OFED 1.0 is ancient by now -- you are much better off just<br>
> using drivers from the standard kernel. If you must use OFED,
then<br>
> OFED 1.2 or even a 1.3 prerelease would be better.<br>
> <br>
> > My question is why there's no synchronization mechanism
for the command <br>
> <br>
> > execution on HCA, can I use "spin_lock" or "sem_wait"
to synchronize <br>
> > between every command?<br>
> <br>
> The HCA firmware allows multiple commands to be queued. The<br>
> dev->cmd.event_sem semaphore is used to limit the number of<br>
> outstanding commands to the HCA's capabilities, and the<br>
> dev->cmd.hcr_mutex mutex is used to serialize the actual writing
of<br>
> commands to the HCA.<br>
> <br>
> There was a mmiowb() added to mthca_cmd_post() fairly recently that<br>
> might fix your problems if you are running on a large SGI Altix system.<br>
> <br>
> - R.<br>
> <br>
> _____________________________________________________________________________<br>
> Scanned by IBM Email Security Management Services powered by MessageLabs.
<br>
> For more information please visit http://www.ers.ibm.com<br>
> _____________________________________________________________________________<br>
> <br>
> <br>
<br>
_____________________________________________________________________________<br>
Scanned by IBM Email Security Management Services powered by MessageLabs.
For more information please visit http://www.ers.ibm.com<br>
_____________________________________________________________________________<br>
</font></tt>
<br>