[libfabric-users] using infiniband on summit

[Hefty, Sean]

> The blurb tell me that - “The ConnectX-5 has two 100 Gbit/s Infiniband ports. Each port
> connects to a different top-of-rack switch. IBM says each switch card supplies a
> compute node with peak bandwidth of 25 GigaBytes/second. The shared PCIe 4.0 x16 port
> can handle a peak of 32 GB/s, so each node coupled has plenty of bandwidth to support
> peak InfiniBand EDR rates.”
> 
> My question is - in order to use the (rxm over verbs) infiniband interface effectively,
> will I need a single endpoint per node (Since I see only one ofi_rxm provider, then I’m
> guessing that a single endpoint will be required if I use that), or do I need to manage
> multiple (say 2, one per port). I’ve never used a card with multiple ports before and
> I’m surprised to see mxl_0/1/2/3 which appears like 4 domains - are these fundamentally
> different - or windows onto the same device in essence. If there is a page on the wiki
> anywhere that discusses this, please point me to it. I’d like to read up on how
> multiple ports are managed.

Each port on the verbs device will show up as a separate OFI domain.  Different OFI domains may point back to the same device.  Note that from the viewpoint of the local SW, there's no guarantee that different ports are connected into the same network or reach the same peer nodes.

> If there were two separate NIC cards on the node, then I’d need two endpoints yes? Will
> libfabric ever take care of that for me?

There is a multi-rail provider that can stripe data across different ports/devices.  However, the benefit of using multiple rails usually doesn't show up except for very large transfers.

> Flyby question : are there any plans for networking libfabric/ucx meetups at BOFs or
> other at SC19?

There's none that I'm aware of at this time.

- Sean