Dear Tom/all<div><br></div><div> I understand the end to end credit based flow control at the link layer where we have a 32 bit Flow control packet being sent for each VL (with FCCL and FCTBS fields) but I fail to understand where this scheme is implemented in the driver. (OFED linux- 1.4 stack, hw-mthca) . I can see a file with a credit table mapped to different credits counts and another that computes the AETH based on this credit table. </div>
<div>1. Is this the place where the flow control packets are formulated? </div><div>2. If yes, I don't see them computing this for each VL. why? If no, is it a mid layer flow control?</div><div>3. And thats why I have this basic question--> is the link layer implemented as part of OFED stack at all? or does it go into the hardware HCA as firmware? As I understand the hardware vendor only provides verbs to communicate with the HCA.</div>
<div><br></div><div>Pardon me if i am bundling you all with a lot with questions. I am new to all this and I am trying my best to understand the stack. </div><div>Thank you,</div><div>Ashwath</div><div><br></div><div><br>
<div class="gmail_quote">On Tue, Aug 11, 2009 at 10:37 PM, Nifty Tom Mitchell <span dir="ltr"><<a href="mailto:niftyompi@niftyegg.com">niftyompi@niftyegg.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div><div></div><div class="h5">On Mon, Aug 10, 2009 at 12:11:22PM -0400, Ashwath Narasimhan wrote:<br>
><br>
> I looked into the infiniband driver files. As I understand, in order to<br>
> limit the data rate we manipulate the credits on either ends. Since the<br>
> number of credits available depends on the receiver's work receive<br>
> queue size, I decided to limit the queue size to say 5 instead of 8192<br>
> (reference---> ipoib.h, IPOIB_MAX_QUEUE_SIZE to say 3 since my higher<br>
> layer protocol is ipoib). I just want to confirm if I am doing the<br>
> right thing?<br>
<br>
</div></div>Data rate is not manipulated by credits.<br>
Credits and queue sizes are different and have different purposes.<br>
<br>
Visit the Infiniband Trade Association web site and grab the IB<br>
specifications to understand some of the hardware level parts.<br>
<br>
<a href="http://www.infinibandta.org/" target="_blank">http://www.infinibandta.org/</a><br>
<br>
InfiniBand offers credit based flow control and given the nature of<br>
modern IB switches and processors a very small credit count can still<br>
result in full data rate. Having said that flow control is the lowest<br>
level throttle in the system. Reducing the credit count forces the<br>
higher levels in the protocol stack to source or sink the data through<br>
the hardware before any more can be delivered. Thus flow control can<br>
simplify the implementation of higher level protocols. It can also be used<br>
to cost reduce or simplify hardware design (smaller hardware buffers).<br>
<br>
The IB specifications are way too long. Start with this FAQ.<br>
<br>
<a href="http://www.mellanox.com/pdf/whitepapers/InfiniBandFAQ_FQ_100.pdf" target="_blank">http://www.mellanox.com/pdf/whitepapers/InfiniBandFAQ_FQ_100.pdf</a><br>
<br>
The IB specification is way too full of optional features. A vendor may<br>
have XYZ working fine and dandy on one card and since it is optional not<br>
at all on another.<br>
<br>
The various queue sizes for the various protocols built on top of<br>
IB establish transfer behavior in keeping with system interrupt,<br>
system process time slice, system kernel activity loads and needs.<br>
It is counter intuitive but in some cases small queues result in<br>
more responsive and agile systems, especially in the presence of errors.<br>
<br>
Since there are often multiple protocols on the IB stack all protocols<br>
will be impacted by credit tinkering. Most vendors know their hardware<br>
so most drivers will have credit related code optimum.<br>
<br>
In the case of TCP/IP the interaction between IB bandwidths&MTU (IPoIB),<br>
ethernet bandwidth&MTU and even localhost (127.0.0.1) bandwidth&MTU can<br>
be "interesting" depending on host names, subnets, routing etc. TCP/IP<br>
has lots of tuning flags well above the IB driver. I see 500+ net.*<br>
sysctl knobs on this system.<br>
<br>
As you change things do make the changes on all the moving parts, benchmark<br>
and keep a log. Since there are multiple IB hardware vendors<br>
it is important to track hardware specifics. "lspci" is a good tool<br>
to gather chip info. With some cards you also need specifics about<br>
the active firmware.<br>
<br>
So go forth (RPN forever) and conquer.<br>
<font color="#888888"><br>
<br>
--<br>
T o m M i t c h e l l<br>
Found me a new hat, now what?<br>
<br>
</font></blockquote></div><br><br clear="all"><br>-- <br>regards,<br>Ashwath<br>
</div>