<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=Content-Type content="text/html; charset=us-ascii">
<META content="MSHTML 6.00.6000.16640" name=GENERATOR></HEAD>
<BODY>
<DIV><FONT face=Arial size=2><SPAN class=118541415-08012009>Hello,
Alex,</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN class=118541415-08012009>I tried to test
IPoIB_CM module (in connected mode, rev. 1797) and found a problem when sending
large ping. </SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN class=118541415-08012009><SPAN
class=118541415-08012009>Settings: MTU_SIZE == 32K (32768), Connected Mode: On,
ping size: 32K </SPAN></SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN class=118541415-08012009>Below is the
description of the problem and some investigations. </SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN class=118541415-08012009>Please, let us know
your suggestions and comments regards this issue; meanwhile, we continue to
debug.</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN
class=118541415-08012009></SPAN></FONT> </DIV>
<DIV><FONT face=Arial size=2><SPAN
class=118541415-08012009></SPAN></FONT> </DIV>
<DIV><FONT face=Arial size=2><SPAN class=118541415-08012009>This ping always
fails and almost always leads both machines to crash.</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN class=118541415-08012009>1. I enlarged the
size of _ipoib_pkt!data to be MAX_CM_PAYLOAD_MTU just to avoid possible memory
corruption on receive side during the debug.</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN class=118541415-08012009>I found that this
struct also used during CM operations and theoretically can receive buffers
larger than 4K (Max MTU size for UD)</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN
class=118541415-08012009></SPAN></FONT> </DIV>
<DIV><FONT face=Arial size=2><SPAN class=118541415-08012009>typedef struct
_ipoib_pkt<BR>{<BR> ipoib_hdr_t hdr;<BR> union
_payload<BR> {</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN class=118541415-08012009>-
uint8_t data[MAX_UD_PAYLOAD_MTU];<BR>+
uint8_t data[MAX_CM_PAYLOAD_MTU];<BR> ipoib_arp_pkt_t arp;<BR> ip_pkt_t ip;</SPAN></FONT></DIV>
<DIV><FONT face=Arial color=#0000ff size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2><SPAN
class=118541415-08012009> } PACK_SUFFIX type;</SPAN></FONT></DIV>
<DIV><FONT face=Arial color=#0000ff size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2><SPAN class=118541415-08012009>} PACK_SUFFIX
ipoib_pkt_t;</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN
class=118541415-08012009></SPAN></FONT> </DIV>
<DIV><FONT face=Arial size=2><SPAN class=118541415-08012009>2. Now the crash was
not reproduced, but ping still didn't work.</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN class=118541415-08012009>That's because
MAX_WRS_PER_MSG was defined as
MAX_CM_PAYLOAD_MTU/MAX_UD_PAYLOAD_MTU.</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN class=118541415-08012009>In our case,
MAX_WRS_PER_MSG was equal to 16 (62520/4092).</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN class=118541415-08012009>In the case of ICMP
packet IPoIB_CM still uses the UD mode; i.e. it tries to fragment the message
into chunks of 'payload_mtu'.</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN class=118541415-08012009>In our case,
params.payload_mtu == 2K. That is, Ping of 32K should be fragmented into 16
chunks.</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN
class=118541415-08012009></SPAN></FONT> </DIV>
<DIV><FONT face=Arial size=2><SPAN class=118541415-08012009>From the
__send_fragments:</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN class=118541415-08012009>seg_len = ( next_sge
> ( p_port->p_adapter->params.payload_mtu - wr_size )
)?<BR> ( p_port->p_adapter->params.payload_mtu -
wr_size ) : next_sge;</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN
class=118541415-08012009></SPAN></FONT> </DIV>
<DIV><FONT face=Arial size=2><SPAN class=118541415-08012009>3. But there's a
following check inside __send_fragments:</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN
class=118541415-08012009></SPAN></FONT> </DIV>
<DIV><FONT face=Arial size=2><SPAN class=118541415-08012009> if(
wr_idx >= ( MAX_WRS_PER_MSG - 1 ) )<BR> return
NDIS_STATUS_RESOURCES;</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN
class=118541415-08012009></SPAN></FONT> </DIV>
<DIV><FONT face=Arial size=2><SPAN class=118541415-08012009>In our case, wr_idx
always reach 15 (i.e. 16 elements), so this check fails.</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN
class=118541415-08012009></SPAN></FONT> </DIV>
<DIV><FONT face=Arial size=2><SPAN class=118541415-08012009>4. And there's yet
another inconsistence:</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN class=118541415-08012009>Not all cards
support 4K MTU (in UD mode). Originally, it was possible to enlarge MTU to 4K by
using IPoIB parameters.</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN class=118541415-08012009>Now consider the
following code:</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN
class=118541415-08012009></SPAN></FONT> </DIV>
<DIV><FONT face=Arial size=2><SPAN
class=118541415-08012009> p_adapter->params.cm_payload_mtu
=<BR> min( MAX_CM_PAYLOAD_MTU, p_adapter->params.payload_mtu
);<BR> p_adapter->params.cm_xfer_block_size =
<BR> p_adapter->params.cm_payload_mtu +
sizeof(eth_hdr_t);<BR> p_adapter->params.payload_mtu =
<BR><EM> min( DEFAULT_PAYLOAD_MTU,
p_adapter->params.payload_mtu);</EM><BR> p_adapter->params.xfer_block_size
= (sizeof(eth_hdr_t) + p_adapter->params.payload_mtu);</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN
class=118541415-08012009></SPAN></FONT> </DIV>
<DIV><FONT face=Arial size=2><SPAN class=118541415-08012009>It means that if you
has card that supports 4K MTU, p_adapter->params.payload_mtu will
nevertheless get only 2K value (and that's still important in UD
mode)</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN
class=118541415-08012009></SPAN></FONT> </DIV>
<DIV><FONT face=Arial size=2><SPAN
class=118541415-08012009>Thanks,</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN
class=118541415-08012009>XaleX</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN
class=118541415-08012009></SPAN></FONT> </DIV></BODY></HTML>