[ewg] Re: [OMPI devel] [ofa-general] Re: OMPI over ofed udapl - bugs opened

[Or Gerlitz]

Jeff Squyres wrote:
> On May 10, 2007, at 9:02 AM, Or Gerlitz wrote:

>> To start with, my hope here is at least to be able play defensive 
>> here, that is convince you that the disadvantages are minor, where 
>> only if this fails, would schedule myself some reading into the 
>> ipoib-cm rfc to dig the advantages.

> I ask about the advantages because OMPI currently treats QP's as 
> bi-directional.  Having OMPI treat them at unidirectional would be a 
> change.  I'm not against such a change, but I think we'd need to be 
> convinced that there are good reasons to do so.  For example, on the 
> surface, it seems like this scheme would simply consume more QPs and 
> potentially more registered memory (and is therefore unattractive).

Indeed you would need two QPs per btl connection, however, for each 
direction you can make the relevant QP consume ~zero resources per the 
other direction, ie on side A:

for the A --> B QP : RX WR num  = 0, RX SG size = 0
for the B --> A QP : TX WR num  = 0, TX SG size = 0

and on side B the other way. I think that IB disallows to have zero len 
WR num so you set it actually to 1. Note that since you use SRQ for 
large jobs you have zero overhead for RX resources and this one TX WR 
overhead for the "RX" connection on each side. This is the only memory 
related overhead since you don't have to allocate any extra buffers over 
what you do now.

Or.