[ofw] [PATCH] Make WSD use new IBAT library

[Fab Tillier]

>> The GetIpNetEntry2 function doesn't take an overlapped, so making it an
>> overlapped operation would be a tad difficult without kicking off a
>> thread and having the thread poll periodically (or some similar
>> concept).
>  There's no notification mechanisms available to know when these tables
> have changed?  Something like NotifyRouteChange2 that could indicate
> that polling at least has a chance to succeed, even if the change was to
> an unrelated route.  I would think even ipoib could generate such an
> event.

NotifyRouteChange2 is callback driven, which messes with the thread model of the caller if they're single threaded.

I suppose IPoIB could add a mechanism so that when an ARP response is received it triggers it.  I don't know how complicated this could be.  Hopefully by removing the PR query from IPoIB it will respond to ARPs faster and we'll never see pending.

>> Pending is related to the resolution.  The IP Helper library will cause
>> an ARP to be sent if there isn't a valid entry for that destination.
>> IPoIB will send the ARP, the recipient will do a PR query to send the
>> response.  I don't know which of the operations actually cause the
>> state to be returned as incomplete, though, just that I saw it come
>> back as such, and wanted to allow MS MPI to go do something else (such
>> as try to connect to someone else if needed) rather than block and poll.
>  This could be a long wait, especially on larger fabrics during job
> start-up.

Yes, and it would be even longer if each connection blocked the process's only thread.

-Fab