[ofiwg] the send credit thread that will not die

[Reese Faucette (rfaucett)]

Agree, I think that's a primary stumbling block in defining how to check for whether a send will succeed or not anyhow.  Perhaps for insight into how to abstract this size, we could start with an inventory of the actual size characteristics we are trying to abstract:

- sockets: # of bytes in sockbuf
- Infiniband: # of pending operations == length of queue
- usNIC: # of pending SGEs == length of queue
- meta-endpoint, constructed from multiple tx/rx contexts: ??

Others with same or different characteristics?

For usNIC, Infiniband, and possibly others a fair question is "why would you ever want to allocate anything less than maximum queue size?"  Host memory constraints is all I can really think of, so from app writer perspective maybe total host memory consumed is all you really care about. (which is perhaps what you were saying when you started talking about "bytes" rather than "credits"?).  

I could see "size" being the total number of size-dependent bytes needed to support a given set of EP charastics (e.g. for usnic it's queue len + some ancillary structures per queue entry, but not the one-time QP struct needed by every endpoint, regardless of size).  The provider would report this "size" for its default (likely maximal, but not necessarily) EP configuration, which the app could then adjust if memory pressure becomes an issue.  Given a different requested size, the provider would make adjustments to queue configuration to optimally make use of the available memory (by adjusting queue depth or per-entry copy buffer size or whatever is best for that provider)

In the default case, the app write should really not have to care about EP size in terms of queue entries or SGEs.  By rolling these into a total "size" in bytes, sockbuf sizes and QPs lengths can be made to smell about the same to the user.

-reese

> -----Original Message-----
> From: ofiwg-bounces at lists.openfabrics.org [mailto:ofiwg-
> bounces at lists.openfabrics.org] On Behalf Of Hefty, Sean
> Sent: Wednesday, October 15, 2014 6:01 PM
> To: ofiwg at lists.openfabrics.org
> Subject: [ofiwg] the send credit thread that will not die
> 
> Rather than focusing on the best way to expose whether the next operation
> will see EAGAIN or not, I think it would be useful to step back to an earlier
> point in an application's life and focus on how an application specifies the
> *size* of a given transmit/receive context.  And define what the size field(s)
> mean.
> 
> - Sean
> 
> 
> Even if an admin specifies a default size, an app should be able to adjust it,
> up to some maximum administrative value.
> _______________________________________________
> ofiwg mailing list
> ofiwg at lists.openfabrics.org
> http://lists.openfabrics.org/mailman/listinfo/ofiwg