[openib-general] Plans for libibverbs 1.0, 1.1 and beyond

[Sayantan Sur]

* On Feb,43 Or Gerlitz<ogerlitz at voltaire.com> wrote :
> Gil Bloch wrote:
> >I believe we should add support for a resize WQ command (as a part of
> >modify QP) to enable changing the WQ size.
> >On a very large scale cluster, with many operating QPs, the work queue
> >memory consumption might be expansive. Thus the MPI implementation
> >should tradeoff for pipelining requests vs. WQ memory consumption. The
> >resize WQ will allow on-demand adaptive WQ setting instead of static
> >allocation of the memory resource, which I believe can increase
> >performance and save memory at the same time. 
> 
> As others pointed, i think such changes should be driver by the 
> combination of
> 	a) ---real--- need of an app
> 	b) support of as much HW vendors as possible
> 
> Did any IB MPI group approached this list expressing a need for this 
> feature?

I had previously posted something along these lines. You can see

http://openib.org/pipermail/openib-general/2005-December/014655.html

> 
> Moreover, here are some more points that might suggest this feature is 
> not needed:
> 
> - why does the size of the cluster relates to how many credits rank A in 
>  an mpi job would have on the connection (QP) with rank B?

For a given number of credits per connection, increasing the number of
connections linearly increases the memory requirement.

The resize QP verb would also allow for resizing the number of max. send
work requests (thus the memory required per connection). It will have
impact both on small & large scale clusters.

> 
> - who said that (say) eight credits per connection are not excellent for 
> an IB MPI?

The optimal number of credits per connection is highly dependent on the
kind of MPI application being used. I don't think any one number is the
optimal for all applications.

> - some MPIs (eg open MPI open connections by demand so only if rank A 
> attempts to send something to rank B then A connects to B

I think resizing QPs is an orthogonal issue to the on-demand connection
model. Even with this model, applications may end up with a large (if
not fully connected) number of QPs. IMHO, this re-sizing QPs will be
beneficial for such scenarios to reduce memory requirement further.

> - what you really might want to resize is an SRQ in case the 
> implementations want to open connections by demand and add more and more 
> QPs to the SRQ, and the work on modify SRQ verb was initiated by Mellanox?

I completely agree ... resizing SRQ would also be helpful.


Thanks,
Sayantan.

> 
> - same for resize CQ - the verb exist and also the need for it.
> 
> Or.
> 
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit 
> http://openib.org/mailman/listinfo/openib-general

-- 
http://www.cse.ohio-state.edu/~surs