[ofiwg] send/recv "credits"

[Hefty, Sean]

> Then you go from
> 
> if (have_sent < can_send) {
>   post()
> }
> 
> To:
> 
> if (failed_send) {
>   if (post(failed_send) == EGAIN)
>       return ...
> 
> if (post(cur_send) == EGAIN)
>   failed_send = cur_send

EAGAIN behaves similar to what an app would need to do if they were using sockets.  I would expect code similar to this:

if (queue_empty()) {
	ret = post(op);
	if (ret == -EAGAIN)
		queue_op(op);
}

If the app needs to track flow control manually, that's a separate issue than the 'credit' discussion being addressed.  I'm trying to handle the case where the send queue is no longer a fixed size that accepts exactly N requests.  ("Credits" is not the right term here.)

Independent from EAGAIN, Does the op_size / iov_size / op_alignment proposal work for apps that want to track send queue usage separate from the provider's tracking?

> I'm sort of confused here, introducing hidden flow control is a
> middleware function - libfabric will need its own private frame
> structure to make that work portably. Is that the intent?

Flow control currently isn't defined within libfabric.  The intent is to expose whatever flow control mechanism a low-level provider (on that sits directly on hardware) may support, but also enable high-level providers that may implement improved flow control support.

A high-level provider may extend the capabilities in other ways, such as adding support for tag matching, expanding atomics, supporting larger message sizes, etc.  So that every app that does need these features doesn't need to implement everything.  Libfabric can layer directly over the HW, or above an abstraction layer, such as PSM or MXM.  Eventually the hope is that HW provides additional capabilities that an app can take advantage of without the apps always needing to change every time every vendor decides to expose some new feature.

> > Actually, a completion queue may use variable sized entries as well.
> 
> It doesn't seem something like that could exist without a hidden
> on-the-wire flow control scheme of some sort (like RNR ACK).
> 
> We already basically have that requirement for many SRQ and XRC type
> applications, so it isn't really a new thing.

I didn't think SRQ or XRC protect against overrunning the remote CQ.