[ewg] Re: [OMPI devel] [ofa-general] Re: OMPI over ofed udapl - bugs opened

[Gleb Natapov]

On Thu, May 10, 2007 at 04:30:27PM +0300, Or Gerlitz wrote:
> Jeff Squyres wrote:
> >On May 10, 2007, at 9:02 AM, Or Gerlitz wrote:
> 
> >>To start with, my hope here is at least to be able play defensive 
> >>here, that is convince you that the disadvantages are minor, where 
> >>only if this fails, would schedule myself some reading into the 
> >>ipoib-cm rfc to dig the advantages.
> 
> >I ask about the advantages because OMPI currently treats QP's as 
> >bi-directional.  Having OMPI treat them at unidirectional would be a 
> >change.  I'm not against such a change, but I think we'd need to be 
> >convinced that there are good reasons to do so.  For example, on the 
> >surface, it seems like this scheme would simply consume more QPs and 
> >potentially more registered memory (and is therefore unattractive).
> 
> Indeed you would need two QPs per btl connection, however, for each 
> direction you can make the relevant QP consume ~zero resources per the 
> other direction, ie on side A:
> 
> for the A --> B QP : RX WR num  = 0, RX SG size = 0
> for the B --> A QP : TX WR num  = 0, TX SG size = 0
> 
> and on side B the other way. I think that IB disallows to have zero len 
> WR num so you set it actually to 1. Note that since you use SRQ for 
> large jobs you have zero overhead for RX resources and this one TX WR 
> overhead for the "RX" connection on each side. This is the only memory 
> related overhead since you don't have to allocate any extra buffers over 
> what you do now.
> 
QP is a limited resource and we already have 2 per connection (and much
more if LMC is in used), so I don't see any reason to use this scheme only
to overcome brain damaged design of iWarp.

--
			Gleb.