<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<META NAME="Generator" CONTENT="MS Exchange Server version 5.5.2654.45">
<TITLE>RE: [openib-general] IPoIB IETF WG presentation updated again</TITLE>
</HEAD>
<BODY>
<BR>
<P><FONT SIZE=2>>-----Original Message-----</FONT>
<BR><FONT SIZE=2>>From: Hal Rosenstock [<A HREF="mailto:halr@voltaire.com">mailto:halr@voltaire.com</A>]</FONT>
<BR><FONT SIZE=2>>Sent: Tuesday, August 03, 2004 5:13 PM</FONT>
</P>
<P><FONT SIZE=2>>Hi Dror,</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>>> It's good to see that MAX_ADDR_LEN has been changed to 32. Does that solve </FONT>
<BR><FONT SIZE=2>>> all the IPoIB ARP related problems for 2.6 kernel ? Can we store all related link </FONT>
<BR><FONT SIZE=2>>> information in this 32 bytes ? What is envisioned to be stored in this 32 bytes - is </FONT>
<BR><FONT SIZE=2>>> it just the QPN+GID, or the entire path info, or the address vector object too ? </FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>>If it holds 32 bytes, then it can hold GID + QPN with 13 bytes still available.</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>>Other information you might want to hold:</FONT>
<BR><FONT SIZE=2>>SL 1 bytes</FONT>
<BR><FONT SIZE=2>>LID 2 bytes</FONT>
<BR><FONT SIZE=2>>MTU (for connected mode) (1 byte)</FONT>
<BR><FONT SIZE=2>>Rate (1 byte)</FONT>
<BR><FONT SIZE=2>>Network Layer</FONT>
<BR><FONT SIZE=2>>Flow Label 3 bytes (20 bits)</FONT>
<BR><FONT SIZE=2>>Hop Limit 1 byte</FONT>
<BR><FONT SIZE=2>>TClass 1 byte</FONT>
<BR><FONT SIZE=2>> </FONT>
</P>
<P><FONT SIZE=2>Source Path bits (1 bytes) ?</FONT>
</P>
<P><FONT SIZE=2>>So all the info for an AV could be stored there. Did I miss something needed ? I didn't </FONT>
<BR><FONT SIZE=2>>double check this but there is still some room left over.</FONT>
</P>
<P><FONT SIZE=2>This is the information for the AV. However, you don't want to create the AV for each packet that you send. Although in Mellanox devices the creation of AVs is relatively inexpensive (compared with other resources like QP, CQ), I don't think that it's the right way to go. I think that the right way to go is to store the AV handle in the ARP table as part of the HW address. And here comes the problem...</FONT></P>
<P><FONT SIZE=2>If you're not notified of ARP table entry invalidation, then you can not really destroy the AV handle. So you have now to maintain some state yourself. And I was wondering if there is a way to solve that. </FONT></P>
<P><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>>> I think that ideally, if a network device can replace the ARP functionality in the kernel </FONT>
<BR><FONT SIZE=2>>> that'll be better. Because this way the IPoIB can get an address resolution request </FONT>
<BR><FONT SIZE=2>>> from the IP stack, handle it by sending an ARP, then SA query for the path record, then </FONT>
<BR><FONT SIZE=2>>> creation of HCA address handle, and then place it in cache and pass back this address </FONT>
<BR><FONT SIZE=2>>> handle. When cache is replaced or expires, IPoIB will destroy the HCA address handle. </FONT>
<BR><FONT SIZE=2>>> If this is not supported, then IPoIB will still need to maintain a shadow table. </FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>>Cloning an AH is probably faster than creating a new one from scratch. (We would need</FONT>
<BR><FONT SIZE=2>>an additional verb for this). How much does this cost ? Is this optimization worth it ?</FONT>
</P>
<P><FONT SIZE=2>In Tavor the creation/modification of AV is relatively inexpensive. It involves writing the AV information to the attached DRAM or to the main memory. For userland application, it may be more complicated in some mode of operation. Writing AV to DRAM involves PIO writes, which you probably want to avoid on the datapath. So, you'd better have an AV ready and persistent for each neighbor, as long as your cache is long enough.</FONT></P>
<P><FONT SIZE=2>BTW, I don't think that modification will almost cost you the same as new creation.</FONT>
</P>
<P><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>> Beyond that, it'll be nice if we could have gotten the IP datagram without the "Ethernet" </FONT>
<BR><FONT SIZE=2>>> header. Currently the IPoIB driver has to chop it, and replace it with the IPoIB encapsulation </FONT>
<BR><FONT SIZE=2>>> header. Anyway, this is just the purity of the protocol stack layering. </FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>>There would need to be another way to identify the various protocols (aka ethertypes) being >carried.</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>Yes. That was what Roland mentioned. Implement hard_header.</FONT>
</P>
<P><FONT SIZE=2>-Dror</FONT>
</P>
</BODY>
</HTML>