[ofa-general] device attributes

[Tom Tucker]

On Wed, 2008-06-04 at 09:28 -0500, Steve Wise wrote:
> Roland/All,
> 
> Should the device attributes (for instance max_send_wr) be the max 
> supported by the HW or the max supported by the OS or something else?
> 

Something else. 

> For instance:  Chelsio's HW can handle very large work queues, but since 
> Linux limits the size of contiguous dma coherent memory allocations, the 
> actual limits are much smaller.

Basing the limit on an OS resource seems arbitrary and dangerous.
Applications using advertised adapter resource limits will unnecessarily
consume the maximum.

>   Which should I be using for the device 
> attributes?
> 

Arbitrary knee jerk == 512. 

However, surveying the current app usage as well as the other
manufacturers advertised limits will make it less arbitrary.

> Also, the chelsio device uses a single work queue to implement the SQ 
> and RQ abstractions.  So the max SQ depth depends on the RQ depth and 
> vice versa.  This leads to device max attributes that aren't that useful.
> 

So the "real" limit is the HW WQ max and therefore max SQ = HW WQ max -
RQ max?  Setting RQ and SQ to 512 solves this "problem." 


> I'm wondering what application writes should glean from these attributes...
> 

I guess you mean "application writer"? 

Here's what I suggest:

- Set the RQ and SQ max to some reasonable default limit (e.g. 512).
- Add an escape hatch by providing module options to override
  the default max.

Tom

> Steve
> 
> _______________________________________________
> general mailing list
> general at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general