<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii">
<META NAME="Generator" CONTENT="MS Exchange Server version 6.5.7653.2">
<TITLE>Infiniband data transfer across different IB drivers</TITLE>
</HEAD>
<BODY>
<!-- Converted from text/rtf format -->
<BR>
<P><FONT SIZE=2 FACE="Arial">Hi, </FONT>
<BR><FONT SIZE=2 FACE="Arial">I am currently experimenting with Infiniband data transfers across two servers with different operating systems</FONT>
<BR><FONT SIZE=2 FACE="Arial">and IB drivers.</FONT>
<BR><FONT SIZE=2 FACE="Arial"> </FONT>
<BR><FONT SIZE=2 FACE="Arial">Server A runs VxWorks 5.5 and uses Mellanox IB driver modules and VAPI interface</FONT>
<BR><FONT SIZE=2 FACE="Arial"> </FONT>
<BR><FONT SIZE=2 FACE="Arial">Server B runs Linux 2.6.x and uses OFED 1.0 drivers and the OFED Verbs API</FONT>
</P>
<P><B><FONT SIZE=2 FACE="Arial">Problem:</FONT></B>
<BR><FONT SIZE=2 FACE="Arial">I have written code (that makes the respective Verbs calls) to setup queue pairs and initialize them with the</FONT>
<BR><FONT SIZE=2 FACE="Arial">destination queue pair number and lid. The connection type is IBV_QPT_RC (Reliable Connection).</FONT>
<BR><FONT SIZE=2 FACE="Arial">The traces seem to confirm that the destination qpn, lid values are correct. The next thing</FONT>
<BR><FONT SIZE=2 FACE="Arial">I try to do is to post send requests on Server A, and receive requests on Server B. I then check the </FONT>
<BR><FONT SIZE=2 FACE="Arial">respective completion queues for any events. The problem is that I do NOT see any completion events on </FONT>
<BR><FONT SIZE=2 FACE="Arial">the receive completion queue for Server B.</FONT>
</P>
<P><B><FONT SIZE=2 FACE="Arial">Questions:</FONT></B>
<BR><FONT SIZE=2 FACE="Arial">- Are these two drivers (Mellanox VAPI and OFED) compatible with each other in the first place?</FONT>
</P>
<P><FONT SIZE=2 FACE="Arial">- Is it possible to verify the two queue pairs are indeed "connected" to each other?</FONT>
</P>
<P><FONT SIZE=2 FACE="Arial">- Can I enable some debug mechanism at the driver level to see what the send/receive requests translate to, and what the underlying</FONT></P>
<P><FONT SIZE=2 FACE="Arial">errors could be (if any)?</FONT>
</P>
<BR>
<P><B><FONT SIZE=2 FACE="Arial">Here is some information about the network that may help</FONT></B><FONT SIZE=2 FACE="Arial">:</FONT>
</P>
<P><B><FONT FACE="Courier">[root@ServerB ~]# ps -elf | grep opensm</FONT></B>
<BR><FONT SIZE=2 FACE="Courier">4 S root 2695 1 0 32 - - 14738 stext Jun14 ? 00:00:00 /usr/local/ofed/bin/opensm -t 200 -g 0</FONT>
<BR><FONT SIZE=2 FACE="Courier">0 S root 12030 11992 0 76 0 - 13981 pipe_w 11:18 pts/1 00:00:00 grep opensm</FONT>
</P>
<P><B><FONT FACE="Courier">[root@ServerB ~]# sminfo</FONT></B>
<BR><FONT SIZE=2 FACE="Courier">sminfo: sm lid 0x1 sm guid 0x2c90200212251, activity count 40926 priority 1 state SMINFO_MASTER 3</FONT>
</P>
<BR>
<P><B><FONT FACE="Courier">[root@ServerB ~]# ibnetdiscover -v</FONT></B>
<BR><FONT SIZE=2 FACE="Courier"> [1] {0002c90200212250}</FONT>
<BR><FONT SIZE=2 FACE="Courier">DR path [0][1] -> new remote ca {00d01c000001010a} portnum 2 lid 0x2-0x2 "ServerA HCA-1 (Topspin HCA)"</FONT>
<BR><FONT SIZE=2 FACE="Courier"> [2] {00d01c000001010a}</FONT>
<BR><FONT SIZE=2 FACE="Courier">#</FONT>
<BR><FONT SIZE=2 FACE="Courier"># Topology file: generated on Fri Jun 15 11:05:52 2007</FONT>
<BR><FONT SIZE=2 FACE="Courier">#</FONT>
<BR><FONT SIZE=2 FACE="Courier"># Max of 1 hops discovered</FONT>
<BR><FONT SIZE=2 FACE="Courier"># Initiated from node 0002c90200212250 port 0002c90200212251</FONT>
</P>
<P><FONT SIZE=2 FACE="Courier">vendid=0xd01c</FONT>
<BR><FONT SIZE=2 FACE="Courier">devid=0x5a44</FONT>
<BR><FONT SIZE=2 FACE="Courier">sysimgguid=0xd01c000001010a</FONT>
<BR><FONT SIZE=2 FACE="Courier">caguid=0xd01c000001010a</FONT>
<BR><FONT SIZE=2 FACE="Courier">Ca 2 "H-00d01c000001010a" # ServerA HCA-1 (Topspin HCA)</FONT>
<BR><FONT SIZE=2 FACE="Courier">[2] "H-0002c90200212250"[1] # lid 2 lmc 0</FONT>
</P>
<P><FONT SIZE=2 FACE="Courier">vendid=0x2c9</FONT>
<BR><FONT SIZE=2 FACE="Courier">devid=0x5a44</FONT>
<BR><FONT SIZE=2 FACE="Courier">sysimgguid=0x2c90200212253</FONT>
<BR><FONT SIZE=2 FACE="Courier">caguid=0x2c90200212250</FONT>
<BR><FONT SIZE=2 FACE="Courier">Ca 2 "H-0002c90200212250" # ServerB HCA-1</FONT>
<BR><FONT SIZE=2 FACE="Courier">[1] "H-00d01c000001010a"[2] # lid 1 lmc 0</FONT>
</P>
<BR>
<P><B><FONT FACE="Courier">[root@ServerB ~]# ibcheckstate -v</FONT></B>
</P>
<P><FONT SIZE=2 FACE="Courier"># Checking Ca: nodeguid 0x00d01c000001010a</FONT>
<BR><FONT SIZE=2 FACE="Courier">Node check lid 2: OK</FONT>
<BR><FONT SIZE=2 FACE="Courier">Port check lid 2 port 2: OK</FONT>
</P>
<P><FONT SIZE=2 FACE="Courier"># Checking Ca: nodeguid 0x0002c90200212250</FONT>
<BR><FONT SIZE=2 FACE="Courier">Node check lid 1: OK</FONT>
<BR><FONT SIZE=2 FACE="Courier">Port check lid 1 port 1: OK</FONT>
</P>
<P><FONT SIZE=2 FACE="Courier">## Summary: 2 nodes checked, 0 bad nodes found</FONT>
<BR><FONT SIZE=2 FACE="Courier">## 2 ports checked, 0 ports with bad state found</FONT>
</P>
<BR>
<P><B><FONT FACE="Courier">[root@ServerB ~]# ibnodes -v</FONT></B>
<BR><FONT SIZE=2 FACE="Courier">Ca : 0x00d01c000001010a ports 2 "ServerA HCA-1 (Topspin HCA)"</FONT>
<BR><FONT SIZE=2 FACE="Courier">Ca : 0x0002c90200212250 ports 2 "ServerB HCA-1"</FONT>
</P>
<BR>
<BR>
<BR>
<P><FONT SIZE=2 FACE="Arial">Please let me know if you need any other information. </FONT>
</P>
<BR>
<P><FONT SIZE=2 FACE="Arial">Thanks in advance,</FONT>
</P>
<P><FONT SIZE=2 FACE="Arial">Kapil</FONT>
</P>
</BODY>
</HTML>