<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=windows-1255">
<META NAME="Generator" CONTENT="MS Exchange Server version 6.5.7226.0">
<TITLE>OFED 1.0 - error while running ib_rdma_bw </TITLE>
</HEAD>
<BODY>
<!-- Converted from text/rtf format -->
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial">Running ib_rdma_bw (eg from the trunk but also with OFED) from time to time outputs the following message:</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"> <FONT SIZE=2 FACE="Arial">server read: Success </FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"> <FONT SIZE=2 FACE="Arial">0/45: Couldn't read remote address </FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial">Looking in the code, line 148 (and actually 142 as well) seems to be buggy:</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial"> 133 struct pingpong_dest * pp_client_exch_dest(int sockfd,</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial"> 134 const struct pingpong_dest *my_dest)</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial"> 135 {</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial"> 136 struct pingpong_dest *rem_dest = NULL;</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial"> 137 char msg[sizeof "0000:000000:000000:00000000:0000000000000000"];</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial"> 138 int parsed;</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial"> 139</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial"> 140 sprintf(msg, "%04x:%06x:%06x:%08x:%016Lx", my_dest->lid, my_dest->qpn,</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial"> 141 my_dest->psn,my_dest->rkey,my_dest->vaddr);</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial"> 142 if (write(sockfd, msg, sizeof msg) != sizeof msg) {</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial"> 143 perror("client write");</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial"> 144 fprintf(stderr, "Couldn't send local address\n");</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial"> 145 goto out;</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial"> 146 }</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial"> 147</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial"> 148 if (read(sockfd, msg, sizeof msg) != sizeof msg) {</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial"> 149 perror("client read");</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial"> 150 fprintf(stderr, "Couldn't read remote address\n");</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial"> 151 goto out;</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial"> 152 }</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial">as read(2) can read less then the max (expected) bytes count, and indeed error is 0 (no error)</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial">when the print is seen.</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial">The below script wouls allow you to easily reproduce it.</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial">At some point, there's also an IB completion with error printed, but it might be realated to the socket handling bug</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial">Or.</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial">SERVER=dill</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial">echo "" </FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial">for i in 16384 32768 65536 131072 262144 524288 1048576 2097152 </FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial">do </FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial">for k in 4 </FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial">do </FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial">ssh $SERVER "/usr/local/ofed/bin/ib_rdma_bw" & </FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial">sleep 5 </FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial">echo $(date) -s = $i -n = $((512*1024*1024/$i)) -t = $k start </FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial">/usr/local/ofed/bin/ib_rdma_bw $SERVER -s $i -n $((512*1024*1024/$i)) -t $k </FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial">echo $(date) -s = $i -n = $((512*1024*1024/$i)) sleeping 3 seconds..... </FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial">sleep 3 </FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial">echo $(date) -s = $i -n = $((512*1024*1024/$i)) end </FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial">echo "" </FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial">wait </FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial">done </FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial">done </FONT></SPAN></P>
</BODY>
</HTML>