[openib-general] using IB on a port without IPoIB running NIC

[Or Gerlitz]

Tang, Changqing wrote:
> We understand that, but we hope to have a connect/accept style IB
> connection setup, without IPoIB involved,

> like HP-UX IT-API(similar to uDAPL without underlying IP support), it
> works with multiple cards.

> Configure 4-5 IP addresses on a single node is kind of silly.

CQ,

Few more thoughts on your "being able to MPI on an IB PORT without an 
IPoIB working NIC" requirement...

Basically, people use IB for both IPC and I/O, where except for SRP, all 
the IB I/O ULPs (both block based: iSER and file based: Lustre, GPFS, 
rNFS) use IP addressing and hence are either coded to the RDMA CM or 
work on top of TCP/IP (iSCSI-TCP, NFS, pFS, etc).

So if the user will not configure IPoIB on this IB port, it will not be 
utilized for I/O.

Now, you mention a use case of 4 cards on a node, I believe that 
typically this would happen on big SMP machines where you **must** use 
all the active IB links for I/O: eg when most of your MPI work is within 
the SMP (128 to 512 ranks) and most of the IB work is for I/O .

I understand (please check and let me know eg about HP 1U offering) that 
all/most nowadays 1U PCI-EX nodes can have at most **one** PCI-EX
card.

Combing the above limitation with the fact that these nodes would run at 
most 16 ranks (eg 8 dual-core CPUs) and that 8 ranks/IB link is a ratio 
that makes sense, we are remained with **2** and not 4-5 NICs to configure.

Oh, and one more thing, 4 IB links per node would make an N node cluster 
to 4N IB end-ports cluster for which you need f(4N) switching IB ports, 
and the specific f(.) might turn the IB deployment over this cluster 
into very expensive one...

Or.