[ofw] Which API to use for IB project

[Sean Hefty]

>1. latency
>2. cpu load
>3. low porting effort to linux compute nodes
>4. implementation effort
>
>Latency is crucial, porting / implementation effort is just nice to have.

For latency, really any of the RDMA APIs (winverbs, IBAL, ND,
libibverbs/librdmacm, uDAPL) will work well.  The data path operations for any
of these calls is relatively thin.

For CPU load, IBAL, libibverbs/librdmacm, and uDAPL allocate additional system
threads.  Again, the overhead is small in each case, but depending on how the
interface is used, you may see additional latency, in particular with event
processing, like CQ notifications.

For portability, uDAPL and libibverbs/librdmacm are available on both Windows
and Linux.  uDAPL abstracts everything, whereas libibverbs does expose some OS
specific structures, like fd's on Linux.  If you keep to the libibverbs API, the
code should be easily portable.  However, if you touch the OS specific
structures, more work would be required to port, but you can also optimize
somewhat for each OS.  This is how uDAPL is coded.

- Sean