[openib-general] [PATCHv2] IPoIB CM Experimental support

[Bernard King-Smith]

"Michael S. Tsirkin" <mst at mellanox.co.il> wrote on 12/16/2006 12:03:28 PM:

> > > > >
> > > > > Tried this patch, it didn't work on ehca. I couldn't change 
> the mode from
> > > > > datagram to connected from /sys/class.
> > > >
> > > > It's wroking as designed in that respect.  ehca does not implement
> > > > srq - without
> > > > srq, there is no way to prepost receive buffers for a 
> resonable number of
> > > > connections without running out of memory.
> > > >
> > > > So it is falling back on datagram mode.
> > > > Talk to ehca guys to implement srq and connected mode will be 
enabled.
> > > Don't remember SRQ is a MUST for UC mode. Does this patch support
> > > devices with SRQ in RC mode?
> > 
> > I don't think the IB HCA Spec requires SRQ support for RC but is an 
optional
> > feature. There are two adapters right now that don't support SRQ 
> which means to
> > use IPoIB-CM on them you should make the use of SRQ an option setting.
> 
> No, adding such "drink up all memory on real clusters but run well 
> on a back to back
> benchmark platform" option does not seem like a good idea to me.
> Rather, we should use UD mode to keep IPoIB scalable on all hardware.

I agree that adapters that don't have SRQ can consume larger amounts of 
memory than those with SRQ ,however, that is not a good reason to prevent 
usage of RC or UC on those adapters. The memory consumption problem with 
any protocol not using SRQ and running over RC or UC is well documented. 
At the OpenFabrics meeting in Tampa one of several themes was that we need 
better IP performance to move into commercial customers and also help our 
current primarily HPC customers, some which are not large numbers of 
endpoints configurations. Even thought other ULP's are available, good IP 
is still the opportunity to getting more customers on IB. 

Not all IB customers we have a large number of endpoint deployments so 
having non SRQ adapters use IPoIB-CM is still important to expanding the 
customer base for IB. You have to let the customer decide how they want to 
tune their system based on the available functions/features. If not you 
don't have equality in potential performance across all HCA's. Some 
guidance on memory consumption would be good, to guide users whether they 
want to run IPoIB-CM without SRQ just like IPoIB-CM will be selectable.

> 
> > I agree
> > that if it is available it should be used for scaling issues probably 
if
> > available automatically set. But I would like to see us at least 
support the
> > current hardware that meets the current SPEC.
> 
> SRQ support is clearly optional. But neither is IPoIB CM support a 
required
> feature. Current code will fall back to datagram mode when SRQ is not
> supported, and since UD support in not optional, all current hardware is 
still
> supported with IPoIB - this patch does not break this.
> 
> -- 
> MST


Bernie King-Smith 
IBM Corporation
Server Group
Cluster System Performance 
wombat2 at us.ibm.com    (845)433-8483
Tie. 293-8483 or wombat2 on NOTES 

"We are not responsible for the world we are born into, only for the world 
we leave when we die.
So we have to accept what has gone before us and work to change the only 
thing we can,
-- The Future." William Shatner
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20061218/2b195a3b/attachment.html>