tests using connected mode IPoIB<div><br></div><div>dmesg from the compute node</div><div><br></div><div><div>cfd-cnsl-0001:~ # dmesg</div><div>ib0: Start path record lookup for fe80:0000:0000:0000:00e0:8111:0100:0091 MTU > 0</div>
<div>ib0: PathRec LID 0x0518 for GID fe80:0000:0000:0000:00e0:8111:0100:0091</div><div>ib0: Created ah ffff810216a8c740</div><div>ib0: created address handle ffff81041c6533c0 for LID 0x0518, SL 0</div><div>ib0: Request connection 0x4a for gid fe80:0000:0000:0000:00e0:8111:0100:0091 qpn 0x404</div>
<div>ib0: REP received.</div><div>ib0: REQ arrived</div><div>ib0: Send unicast ARP to 0518</div><div>ib0: Send unicast ARP to 0518</div><div>ib0: Send unicast ARP to 0518</div><div>ib0: Send unicast ARP to 0518</div><div>
ib0: Send unicast ARP to 0518</div><div>ib0: Start path record lookup for fe80:0000:0000:0000:00e0:8111:0100:007d MTU > 0</div><div>ib0: PathRec LID 0x0161 for GID fe80:0000:0000:0000:00e0:8111:0100:007d</div><div>ib0: Created ah ffff810216a8c580</div>
<div>ib0: created address handle ffff81041e564d40 for LID 0x0161, SL 0</div><div>ib0: Send unicast ARP to 0161</div><div>ib0: REQ arrived</div><div>ib0: Send unicast ARP to 0161</div><div>ib0: REQ arrived</div><div>ib0: Send unicast ARP to 0161</div>
<div>ib0: Send unicast ARP to 0518</div><div>ib0: REQ arrived</div><div>ib0: Send unicast ARP to 0518</div><div>ib0: Send unicast ARP to 0518</div><div>ib0: Send unicast ARP to 0161</div><div>ib0: REQ arrived</div><div>ib0: DREQ received.</div>
<div>ib0: CM error 9.</div><div>ib0: Destroy active connection 0x4a head 0x19680 tail 0x19680</div><div><br></div><div><br></div><div>dmesg from the io node</div><div><br></div><div><div>[root@cfd-io-0001 ~]# dmesg</div><div>
ib0: Start path record lookup for fe80:0000:0000:0000:0002:c903:0000:c36d MTU > 0</div><div>ib0: PathRec LID 0x0384 for GID fe80:0000:0000:0000:0002:c903:0000:c36d</div><div>ib0: Created ah ffff810126e93500</div><div>ib0: created address handle ffff81012b98cc80 for LID 0x0384, SL 0</div>
<div>ib0: Send unicast ARP to 0384</div><div>ib0: REQ arrived</div><div>ib0: Request connection 0x10406 for gid fe80:0000:0000:0000:0002:c903:0000:c36d qpn 0x48</div><div>ib0: REP received.</div><div>ib0: Send unicast ARP to 0384</div>
<div>ib0: Send unicast ARP to 0384</div><div>ib0: Send unicast ARP to 0384</div><div>ib0: Send unicast ARP to 0384</div><div>ib0: Send unicast ARP to 0384</div><div>ib0: Send unicast ARP to 045e</div><div>ib0: REQ arrived</div>
<div>ib0: neigh_destructor for bonding device: ib0</div><div>ib0: neigh_cleanup for 000048 fe80:0000:0000:0000:0002:c903:0000:c36d</div><div>ib0: Reap connection for gid fe80:0000:0000:0000:0002:c903:0000:c36d</div><div>ib0: Destroy active connection 0x10406 head 0x12455 tail 0x12455</div>
<div>ib0: Request connection 0x30406 for gid fe80:0000:0000:0000:0002:c903:0000:cad1 qpn 0x48</div><div>ib0: REP received.</div><div>ib0: Request connection 0x30407 for gid fe80:0000:0000:0000:0002:c903:0000:c36d qpn 0x48</div>
<div>ib0: REP received.</div><div>ib0: neigh_destructor for bonding device: ib0</div><div>ib0: neigh_cleanup for 000048 fe80:0000:0000:0000:0002:c903:0000:cad1</div><div>ib0: Reap connection for gid fe80:0000:0000:0000:0002:c903:0000:cad1</div>
<div>ib0: Destroy active connection 0x30406 head 0x8 tail 0x8</div><div>ib0: Request connection 0x40406 for gid fe80:0000:0000:0000:0002:c903:0000:cad1 qpn 0x48</div><div>ib0: REP received.</div><div>ib0: Send unicast ARP to 0384</div>
<div>ib0: Send unicast ARP to 0384</div><div>ib0: neigh_destructor for bonding device: ib0</div><div>ib0: neigh_cleanup for 000048 fe80:0000:0000:0000:0002:c903:0000:cad1</div><div>ib0: Reap connection for gid fe80:0000:0000:0000:0002:c903:0000:cad1</div>
<div>ib0: Destroy active connection 0x40406 head 0x8 tail 0x8</div><div>ib0: neigh_destructor for bonding device: ib0</div><div>ib0: neigh_cleanup for 000048 fe80:0000:0000:0000:0002:c903:0000:c36d</div><div>ib0: Reap connection for gid fe80:0000:0000:0000:0002:c903:0000:c36d</div>
<div>ib0: Destroy active connection 0x30407 head 0x3 tail 0x3</div><div>ib0: Send unicast ARP to 045e</div><div>ib0: REQ arrived</div><div>ib0: Request connection 0x80405 for gid fe80:0000:0000:0000:0002:c903:0000:cad1 qpn 0x48</div>
<div>ib0: REQ arrived</div><div>ib0: REP received.</div><div>ib0: Send unicast ARP to 045e</div><div>ib0: REQ arrived</div><div>ib0: Send unicast ARP to 045e</div><div><br></div></div><br><div class="gmail_quote">On Thu, Jul 10, 2008 at 9:21 AM, Michael Di Domenico <<a href="mailto:mdidomenico4@gmail.com">mdidomenico4@gmail.com</a>> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">tests using datagram IPoIB (non-connected mode)<div><br></div><div>dmesg from the compute node</div><div><div>cfd-cnsl-0001:~ # dmesg</div>
<div>ib0: Send unicast ARP to 0518</div><div>ib0: Send unicast ARP to 0518</div><div>
ib0: Send unicast ARP to 0518</div><div>ib0: Send unicast ARP to 0518</div><div>ib0: Start path record lookup for fe80:0000:0000:0000:00e0:8111:0100:007d MTU > 1024</div><div>ib0: PathRec LID 0x0161 for GID fe80:0000:0000:0000:00e0:8111:0100:007d</div>
<div>ib0: Created ah ffff81042063dc80</div><div>ib0: created address handle ffff8102206144c0 for LID 0x0161, SL 0</div><div>ib0: Send unicast ARP to 0161</div><div>ib0: Send unicast ARP to 0161</div><div>ib0: Send unicast ARP to 0161</div>
<div>ib0: Send unicast ARP to 0518</div><div>ib0: Send unicast ARP to 0518</div><div>ib0: Send unicast ARP to 0518</div><div>ib0: Send unicast ARP to 0161</div><div class="Ih2E3d"><div>ib0: neigh_destructor for bonding device: ib0</div>
</div><div>
ib0: neigh_cleanup for 000404 fe80:0000:0000:0000:00e0:8111:0100:0091</div><div><br></div><div><br></div><div>dmesg from the IO node</div><div><br></div><div><div>[root@cfd-io-0001 ~]# dmesg</div><div class="Ih2E3d"><div>
ib0: Send unicast ARP to 0384</div>
<div>ib0: Send unicast ARP to 0384</div></div><div>ib_mthca 0000:07:00.0: too many gathers</div><div>ib0: post_send failed</div><div>ib_mthca 0000:07:00.0: opcode invalid</div><div>ib0: post_send failed</div><div>ib_mthca 0000:07:00.0: opcode invalid</div>
<div>ib0: post_send failed</div><div>ib_mthca 0000:07:00.0: opcode invalid</div><div>ib0: post_send failed</div><div>ib_mthca 0000:07:00.0: opcode invalid</div><div>ib0: post_send failed</div><div>ib_mthca 0000:07:00.0: opcode invalid</div>
<div>ib0: post_send failed</div><div>ib_mthca 0000:07:00.0: opcode invalid</div><div>ib0: post_send failed</div><div>ib_mthca 0000:07:00.0: opcode invalid</div><div>ib0: post_send failed</div><div class="Ih2E3d"><div>ib0: Send unicast ARP to 0384</div>
</div><div>ib_mthca 0000:07:00.0: opcode invalid</div><div>ib0: post_send failed</div><div class="Ih2E3d"><div>ib0: Send unicast ARP to 045e</div><div>ib0: REQ arrived</div><div>ib0: Send unicast ARP to 0384</div></div><div>
ib0: Send unicast ARP to 045e</div>
<div>ib_mthca 0000:07:00.0: opcode invalid</div><div>ib0: post_send failed</div><div class="Ih2E3d"><div>ib0: Send unicast ARP to 0384</div></div><div class="Ih2E3d"><div>ib0: neigh_destructor for bonding device: ib0</div>
<div>ib0: neigh_cleanup for 000048 fe80:0000:0000:0000:0002:c903:0000:cad1</div>
</div><div class="Ih2E3d"><div>ib0: neigh_destructor for bonding device: ib0</div><div>ib0: neigh_cleanup for 000048 fe80:0000:0000:0000:0002:c903:0000:c36d</div></div><div class="Ih2E3d"><div>ib0: neigh_destructor for bonding device: ib0</div>
<div>ib0: neigh_cleanup for 000048 fe80:0000:0000:0000:0002:c903:0000:cad1</div>
</div><div class="Ih2E3d"><div>ib0: REQ arrived</div><div>ib0: neigh_destructor for bonding device: ib0</div></div><div>ib0: neigh_cleanup for 000048 fe80:0000:0000:0000:0002:c903:0000:cad1</div><div>ib_mthca 0000:07:00.0: opcode invalid</div>
<div>ib0: post_send failed</div><div class="Ih2E3d">
<div>ib0: neigh_destructor for bonding device: ib0</div><div>ib0: neigh_cleanup for 000048 fe80:0000:0000:0000:0002:c903:0000:c36d</div></div><div>ib_mthca 0000:07:00.0: opcode invalid</div><div>ib0: post_send failed</div>
<div>
ib_mthca 0000:07:00.0: opcode invalid</div><div>ib0: post_send failed</div><div class="Ih2E3d"><div>ib0: Send unicast ARP to 0384</div></div><div class="Ih2E3d"><div>ib0: neigh_destructor for bonding device: ib0</div><div>
ib0: neigh_cleanup for 000048 fe80:0000:0000:0000:0002:c903:0000:c36d</div>
</div><div class="Ih2E3d"><div>ib0: Send unicast ARP to 045e</div></div><div class="Ih2E3d"><div>ib0: neigh_destructor for bonding device: ib0</div><div>ib0: neigh_cleanup for 000048 fe80:0000:0000:0000:0002:c903:0000:cad1</div>
<div><br></div></div></div><div><div></div><div class="Wj3C7c"><br><div class="gmail_quote">
On Thu, Jul 10, 2008 at 8:33 AM, Eli Cohen <<a href="mailto:eli@dev.mellanox.co.il" target="_blank">eli@dev.mellanox.co.il</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><div></div><div>On Thu, Jul 10, 2008 at 07:57:30AM -0400, Michael Di Domenico wrote:<br>
> maybe i spoke too soon, so more output came, i thought it was done<br>
> ib0: mtu > 2044 will cause multicast packet drops.<br>
> eth5: no IPv6 routers present<br>
> ib0: Send unicast ARP to 0384<br>
> ib0: REQ arrived<br>
> ib0: Request connection 0x2c0406 for gid<br>
> fe80:0000:0000:0000:0002:c903:0000:c36d qpn 0x48<br>
> ib0: REP received.<br>
> ib0: Send unicast ARP to 0384<br>
> ib0: Send unicast ARP to 045e<br>
> ib0: REQ arrived<br>
> ib0: Send unicast ARP to 0384<br>
> ib0: Send unicast ARP to 0384<br>
> ib0: Send unicast ARP to 0384<br>
> ib0: Send unicast ARP to 045e<br>
> ib0: REQ arrived<br>
> ib0: neigh_destructor for bonding device: ib0<br>
> ib0: neigh_cleanup for 000048 fe80:0000:0000:0000:0002:c903:0000:c36d<br>
> ib0: Reap connection for gid fe80:0000:0000:0000:0002:c903:0000:c36d<br>
> ib0: Destroy active connection 0x2c0406 head 0x22644 tail 0x22644<br>
> ib0: Request connection 0x2f0406 for gid<br>
> fe80:0000:0000:0000:0002:c903:0000:cad1 qpn 0x48<br>
> ib0: REP received.<br>
> ib0: neigh_destructor for bonding device: ib0<br>
> ib0: neigh_cleanup for 000048 fe80:0000:0000:0000:0002:c903:0000:cad1<br>
> ib0: Reap connection for gid fe80:0000:0000:0000:0002:c903:0000:cad1<br>
> ib0: Destroy active connection 0x2f0406 head 0x4 tail 0x4<br>
<br>
<br>
</div></div>I see you're working in connected mode. Can you please do the<br>
follwoing:<br>
<br>
1. clear dmesg: dmesg -c<br>
2. run again, then send all the output of dmesg<br>
<br>
Do this for both connected and datagram modes.<br>
<br>
Thanks.<br>
<br>
</blockquote></div><br></div></div></div>
</blockquote></div><br></div>