[openib-general] [ANNOUNCE] Initial trunk checkin of ISERinitiator

[Grant Grundler]

On Wed, Aug 24, 2005 at 11:58:18AM +0300, Guy German wrote:
> What about the SRP implementation, which polls the cq in the cq's upcall
> context (interrupt in mthca). I guess that it improves performance, and
> if it is good enough for the SRP (and linux) I think it will be good for
> the iSER *initiator* as well.

yes - that's probably a fair assumption for now.

> The thing is - even if I find it reasonable to do it this way (after
> calculating the estimated amount of time spent in the ISR), I am
> not sure it is a good scalable solution, because there can be several
> initiators and hcas per machine.

Yup - but you don't need to estimate. It can be precisely measured.
See "get_cycles()" and maybe the patch that Christoph Lameter
posted 5-6 monthes ago:
	http://www.gelato.unsw.edu.au/archives/linux-ia64/0504/13877.html

(My advice is to read the whole thread to be aware of the issues
with this patch...but it should be good enough for the above
measurements.)


FWIW, netperf TCP_STREAM over SDP was not close to saturating a single
CPU (1.5Ghz IA64). Two HCAs would probably saturate a single CPU.
While I would hope it never happens that two HCAs both have their MSI-X
interrupts pointed at the same CPU, I'm certain it could happen.
One could argue the sys admin will need to rebalance the interrupt load
manually by reassigning CPUs via /proc/irq/*/smp_affinity.

> The fact that the primitiveness of linux is not a major issue, makes it
> harder to decide on the right way to go here... 

preemtive is a major issue for some uses. But I'm skeptical it is for
the initial clusters I expect people will use RDMA for. So I'm not
going to worry about it for now. There are more important issues.

hth,
grant